Video encoding and decoding

ABSTRACT

A method comprising: encoding pictures into a bitstream, the bitstream comprising at least two scalability layers, pictures being associated with access units and pictures being associated individually with one of the at least two scalability layers; indicating in the bitstream inter-layer prediction dependencies, indicative of direct reference layers, if any, of a first scalability layer and indirect reference layers, if any, of the first scalability layer; selecting an earlier picture in decoding order as a basis for deriving picture order count (POC) related variables for a current picture based on a pre-defined algorithm, the current picture being associated with a current scalability layer, wherein the earlier picture is the closest preceding picture, in decoding order, to the current picture among a set of pictures that are associated with the current scalability layer or any direct or indirect reference layer of the current scalability layer.

TECHNICAL FIELD

The present application relates generally to encoding and decoding ofdigital video material. In particular, the present application relatesto scalable coding.

BACKGROUND

This section is intended to provide a background or context to theinvention that is recited in the claims. The description herein mayinclude concepts that could be pursued, but are not necessarily onesthat have been previously conceived or pursued. Therefore, unlessotherwise indicated herein, what is described in this section is notprior art to the description and claims in this application and is notadmitted to be prior art by inclusion in this section.

A video coding system may comprise an encoder that transforms an inputvideo into a compressed representation suited for storage/transmissionand a decoder that can uncompress the compressed video representationback into a viewable form. The encoder may discard some information inthe original video sequence in order to represent the video in a morecompact form, for example, to enable the storage/transmission of thevideo information at a lower bitrate than otherwise might be needed.

SUMMARY

Some embodiments provide a method for encoding and decoding videoinformation. In some embodiments there is provided methods, apparatusesand computer program products for video coding.

Various aspects of examples of the invention are provided in thedetailed description.

According to a first aspect, there is provided a method comprising:

-   -   encoding pictures into a bitstream, the bitstream comprising at        least two scalability layers and pictures being associated with        access units;    -   selecting an earlier picture in decoding order as a basis for        deriving picture order count (POC) related variables for a        current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

According to an embodiment, the earlier picture is required for decodingof the current picture or the earlier is required to be present in thebitstream by bitstream conformance constraints.

According to an embodiment, the earlier picture is selected from anydirect or indirect reference layer of the layer of the current picture.

According to an embodiment, the earlier picture is selected to be theprevious picture, in decoding order, required to be present in thebitstream, such as a base-layer TRAP picture.

According to an embodiment, the method further comprises:

-   -   encoding at least one POC related syntax element into the        bitstream on the basis of the POC related variables of said        earlier picture defined as a POC reference picture.

According to an embodiment, the method further comprises:

-   -   using the POC reference picture in DeltaPocVal derivation of POC        reset approach for decrementing the POC values of the picture in        a decoded picture buffer.

According to an embodiment, the method further comprises:

-   -   determining layer-wise POC approach as a layer tree including an        independent layer and at least one dependent layer, wherein a        mapping of layers to the layer tree is obtained from the        dependency information between layers encoded into the        bitstream.

According to an embodiment, the method further comprises:

-   -   determining characteristics, including layer identifier(s),        temporal sub-layer identifier(s), and/or picture types, of        pictures on which the current picture may depend on in        prediction; and    -   selecting the previous picture, in decoding order, fulfilling        the characteristics as the POC reference picture.

According to an embodiment, NOLS pictures are indicated in the bitstreamor inferred from sequence-level information, wherein the POC referencepictures are selected among pictures that are not NOLS pictures.

According to an embodiment, the POC reference picture is selected amongpictures that are no CL-RAS pictures.

According to a second aspect, there is provided an apparatus comprisingat least one processor and at least one memory, said at least one memorystored with code thereon, which when executed by said at least oneprocessor, causes an apparatus to perform at least the following:

-   -   encode pictures into a bitstream, the bitstream comprising at        least two scalability layers and pictures being associated with        access units;    -   select an earlier picture in decoding order as a basis for        deriving picture order count (POC) related variables for a        current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

According to a third aspect, there is provided a computer programproduct embodied on a non-transitory computer readable medium,comprising computer program code configured to, when executed on atleast one processor, cause an apparatus or a system to:

-   -   encode pictures into a bitstream, the bitstream comprising at        least two scalability layers and pictures being associated with        access units;    -   select an earlier picture in decoding order as a basis for        deriving picture order count (POC) related variables for a        current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

According to a fourth embodiment there is provided an apparatuscomprising a video encoder configured for encoding a bitstreamcomprising an image sequence, the video encoder comprising

-   -   means for encoding pictures into a bitstream, the bitstream        comprising at least two scalability layers and pictures being        associated with access units;    -   means for selecting an earlier picture in decoding order as a        basis for deriving picture order count (POC) related variables        for a current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

According to a fifth embodiment there is provided a video encoderconfigured for

-   -   encoding pictures into a bitstream, the bitstream comprising at        least two scalability layers and pictures being associated with        access units;    -   selecting an earlier picture in decoding order as a basis for        deriving picture order count (POC) related variables for a        current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

According to a sixth aspect there is provided a method comprising:

-   -   decoding pictures from a bitstream, the bitstream comprising at        least two scalability layers and pictures being associated with        access units;    -   selecting an earlier picture in decoding order as a basis for        deriving picture order count (POC) related variables for a        current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

According to an embodiment, the earlier picture is required for decodingof the current picture or the earlier is required to be present in thebitstream by bitstream conformance constraints.

According to an embodiment, the earlier picture is any direct orindirect reference layer of the layer of the current picture.

According to an embodiment, the earlier picture is the previous picture,in decoding order, required to be present in the bitstream, such as abase-layer IRAP picture.

According to an embodiment, the method further comprises:

-   -   decoding at least one POC related syntax element from the        bitstream on the basis of the POC related variables of said        earlier picture defined as a POC reference picture.

According to an embodiment, the method further comprises:

-   -   using the POC reference picture in DeltaPocVal derivation of POC        reset approach for decrementing the POC values of the picture in        a decoded picture buffer.

According to an embodiment, the method further comprises:

-   -   determining layer-wise POC approach as a layer tree including an        independent layer and at least one dependent layer, wherein a        mapping of layers to the layer tree is obtained from the        dependency information between layers encoded into the        bitstream.

According to an embodiment, the method further comprises:

-   -   determining characteristics, including layer identifier(s),        temporal sub-layer identifier(s), and/or picture types, of        pictures on which the current picture may depend on in        prediction; and    -   selecting the previous picture, in decoding order, fulfilling        the characteristics as the POC reference picture.

According to an embodiment, NOLS pictures are indicated in the bitstreamor inferred from sequence-level information, wherein the POC referencepictures are selected among pictures that are not NOLS pictures.

According to an embodiment, the POC reference picture is selected amongpictures that are no CL-RAS pictures.

According to a seventh aspect, there is provided an apparatus comprisingat least one processor and at least one memory, said at least one memorystored with code thereon, which when executed by said at least oneprocessor, causes an apparatus to perform at least the following:

-   -   decode pictures from a bitstream, the bitstream comprising at        least two scalability layers and pictures being associated with        access units;    -   select an earlier picture in decoding order as a basis for        deriving picture order count (POC) related variables for a        current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

According to an eighth aspect, there is provided a computer programproduct embodied on a non-transitory computer readable medium,comprising computer program code configured to, when executed on atleast one processor, cause an apparatus or a system to:

-   -   decode pictures from a bitstream, the bitstream comprising at        least two scalability layers and pictures being associated with        access units;    -   select an earlier picture in decoding order as a basis for        deriving picture order count (POC) related variables for a        current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

According to a ninth aspect there is provided an apparatus comprising avideo decoder configured for decoding a bitstream comprising an imagesequence, the video decoder comprising

-   -   means for decoding pictures from a bitstream, the bitstream        comprising at least two scalability layers and pictures being        associated with access units;    -   means for selecting an earlier picture in decoding order as a        basis for deriving picture order count (POC) related variables        for a current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

According to a tenth aspect there is provided a video decoder configuredfor

-   -   decoding pictures from a bitstream, the bitstream comprising at        least two scalability layers and pictures being associated with        access units;    -   selecting an earlier picture in decoding order as a basis for        deriving picture order count (POC) related variables for a        current picture based on a pre-defined algorithm,    -   wherein the earlier picture is such that it is required to be        present in a bitstream.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of example embodiments of the presentinvention, reference is now made to the following descriptions taken inconnection with the accompanying drawings in which:

FIG. 1 shows a block diagram of a video coding system according to anembodiment;

FIG. 2 shows an apparatus for video coding according to an embodiment;

FIG. 3 shows an arrangement for video coding comprising a plurality ofapparatuses;

FIG. 4 shows an example of scalable video coding;

FIG. 5 shows an example of POC reset approach;

FIG. 6 shows an example of layer-wise POC approach;

FIG. 7a shows an example of layer-wise POC approach, where a layer-wisedelta POC value is indicated;

FIG. 7b an example of layer-wise POC approach, where a layer-wise POCLSB value is indicated;

FIG. 8 shows an example where the POC reset approach becomes vulnerableto picture loss;

FIG. 9 shows an example where the layer-wise POC fails due to layerdown-switching;

FIG. 10 shows a flow chart of an encoding method according to anembodiment;

FIG. 11 shows a flow chart of a decoding method according to anembodiment;

FIG. 12 shows an example of layer-tree POC signaling and derivationaccording to an embodiment;

FIG. 13 shows an example of loss resilience in layer-tree POC approachaccording to an embodiment;

FIG. 14 shows an example of POC reference picture residing in adifferent access unit for an enhancement layer picture according to anembodiment;

FIG. 15 shows an example where base-layer IDR picture withcross_layer_bla_flag equal to 1 resets POC values according to anembodiment;

FIG. 16 shows an example where POC and layer-tree POC derivation ofindependent layers reside in different access units according to anembodiment;

FIG. 17 shows an example of layer-tree POC derivation according to anembodiment;

FIG. 18 shows an example of layer down-switching with the layer-tree POCapproach according to an embodiment;

FIG. 19 shows an example of layer-wise start-up process with thelayer-tree POC approach according to an embodiment;

FIG. 20 shows an example of an adaptive resolution change bitstreamaccording to an embodiment; and

FIG. 21 shows another example of an adaptive resolution change bitstreamaccording to an embodiment.

DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS

In the following, several embodiments of the invention will be describedin the context of one video coding arrangement. It is to be noted,however, that the invention is not limited to this particulararrangement. In fact, the different embodiments have applications widelyin any environment where improvement of scalable and/or multiview videocoding is required. For example, the invention may be applicable tovideo coding systems like streaming systems, DVD players, digitaltelevision receivers, personal video recorders, systems and computerprograms on personal computers, handheld computers and communicationdevices, as well as network elements such as transcoders and cloudcomputing arrangements where video data is handled.

FIG. 1 shows a block diagram of a video coding system according to anexample embodiment as a schematic block diagram of an exemplaryapparatus or electronic device 50, which may incorporate a codecaccording to an embodiment of the invention. FIG. 2 shows a layout of anapparatus according to an example embodiment. The elements of FIGS. 1and 2 will be explained next.

The electronic device 50 may for example be a mobile terminal or userequipment of a wireless communication system. However, it would beappreciated that embodiments of the invention may be implemented withinany electronic device or apparatus which may require encoding anddecoding or encoding or decoding video images.

The apparatus 50 may comprise a housing 30 for incorporating andprotecting the device. The apparatus 50 further may comprise a display32 in the form of a liquid crystal display. In other embodiments of theinvention the display may be any suitable display technology suitable todisplay an image or video. The apparatus 50 may further comprise akeypad 34. In other embodiments of the invention any suitable data oruser interface mechanism may be employed. For example the user interfacemay be implemented as a virtual keyboard or data entry system as part ofa touch-sensitive display. The apparatus may comprise a microphone 36 orany suitable audio input which may be a digital or analogue signalinput. The apparatus 50 may further comprise an audio output devicewhich in embodiments of the invention may be any one of: an earpiece 38,speaker, or an analogue audio or digital audio output connection. Theapparatus 50 may also comprise a battery 40 (or in other embodiments ofthe invention the device may be powered by any suitable mobile energydevice such as solar cell, fuel cell or clockwork generator). Theapparatus may further comprise a camera 42 capable of recording orcapturing images and/or video. In some embodiments the apparatus 50 mayfurther comprise an infrared port for short range line of sightcommunication to other devices. In other embodiments the apparatus 50may further comprise any suitable short range communication solutionsuch as for example a Bluetooth wireless connection or a USB/firewirewired connection.

The apparatus 50 may comprise a controller 56 or processor forcontrolling the apparatus 50. The controller 56 may be connected tomemory 58 which in embodiments of the invention may store both data inthe form of image and audio data and/or may also store instructions forimplementation on the controller 56. The controller 56 may further beconnected to codec circuitry 54 suitable for carrying out coding anddecoding of audio and/or video data or assisting in coding and decodingcarried out by the controller.

The apparatus 50 may further comprise a card reader 48 and a smart card46, for example a UICC and UICC reader for providing user informationand being suitable for providing authentication information forauthentication and authorization of the user at a network.

The apparatus 50 may comprise radio interface circuitry 52 connected tothe controller and suitable for generating wireless communicationsignals for example for communication with a cellular communicationsnetwork, a wireless communications system or a wireless local areanetwork. The apparatus 50 may further comprise an antenna 44 connectedto the radio interface circuitry 52 for transmitting radio frequencysignals generated at the radio interface circuitry 52 to otherapparatus(es) and for receiving radio frequency signals from otherapparatus(es).

In some embodiments of the invention, the apparatus 50 comprises acamera capable of recording or detecting individual frames which arethen passed to the codec 54 or controller for processing. In someembodiments of the invention, the apparatus may receive the video imagedata for processing from another device prior to transmission and/orstorage. In some embodiments of the invention, the apparatus 50 mayreceive either wirelessly or by a wired connection the image forcoding/decoding.

FIG. 3 shows an arrangement for video coding comprising a plurality ofapparatuses, networks and network elements according to an exampleembodiment. With respect to FIG. 3, an example of a system within whichembodiments of the present invention can be utilized is shown. Thesystem 10 comprises multiple communication devices which can communicatethrough one or more networks. The system 10 may comprise any combinationof wired or wireless networks including, but not limited to a wirelesscellular telephone network (such as a GSM, UMTS, CDMA network etc), awireless local area network (WLAN) such as defined by any of the IEEE802.x standards, a Bluetooth personal area network, an Ethernet localarea network, a token ring local area network, a wide area network, andthe Internet.

The system 10 may include both wired and wireless communication devicesor apparatus 50 suitable for implementing embodiments of the invention.For example, the system shown in FIG. 3 shows a mobile telephone network11 and a representation of the internet 28. Connectivity to the internet28 may include, but is not limited to, long range wireless connections,short range wireless connections, and various wired connectionsincluding, but not limited to, telephone lines, cable lines, powerlines, and similar communication pathways.

The example communication devices shown in the system 10 may include,but are not limited to, an electronic device or apparatus 50, acombination of a personal digital assistant (PDA) and a mobile telephone14, a PDA 16, an integrated messaging device (IMD) 18, a desktopcomputer 20, a notebook computer 22. The apparatus 50 may be stationaryor mobile when carried by an individual who is moving. The apparatus 50may also be located in a mode of transport including, but not limitedto, a car, a truck, a taxi, a bus, a train, a boat, an airplane, abicycle, a motorcycle or any similar suitable mode of transport.

Some or further apparatuses may send and receive calls and messages andcommunicate with service providers through a wireless connection 25 to abase station 24. The base station 24 may be connected to a networkserver 26 that allows communication between the mobile telephone network11 and the internet 28. The system may include additional communicationdevices and communication devices of various types. The communicationdevices may communicate using various transmission technologiesincluding, but not limited to, code division multiple access (CDMA),global systems for mobile communications (GSM), universal mobiletelecommunications system (UMTS), time divisional multiple access(TDMA), frequency division multiple access (FDMA), transmission controlprotocol-internet protocol (TCP-IP), short messaging service (SMS),multimedia messaging service (MMS), email, instant messaging service(IMS), Bluetooth, IEEE 802.11 and any similar wireless communicationtechnology. A communications device involved in implementing variousembodiments of the present invention may communicate using various mediaincluding, but not limited to, radio, infrared, laser, cableconnections, and any suitable connection.

The H.264/AVC standard was developed by the Joint Video Team (JVT) ofthe Video Coding Experts Group (VCEG) of the TelecommunicationsStandardization Sector of International Telecommunication Union (ITU-T)and the Moving Picture Experts Group (MPEG) of InternationalOrganisation for Standardization (ISO)/International ElectrotechnicalCommission (IEC). The H.264/AVC standard is published by both parentstandardization organizations, and it is referred to as ITU-TRecommendation H.264 and ISO/IEC International Standard 14496-10, alsoknown as MPEG-4 Part 10 Advanced Video Coding (AVC). There have beenmultiple versions of the H.264/AVC standard, integrating new extensionsor features to the specification. These extensions include ScalableVideo Coding (SVC) and Multiview Video Coding (MVC).

The H.265/HEVC standard was developed by the Joint Collaborative Team onVideo Coding (JCT-VC) of VCEG and MPEG. The H.265/HEVC standard will bepublished by both parent standardization organizations, and is referredto as ITU-T Recommendation H.265 and ISO/IEC International Standard23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding(HEVC). There are currently ongoing standardization projects to developextensions to H.265/HEVC, including scalable, multiview,three-dimensional, and fidelity range extensions.

A scalable video codec for quality scalability (also known asSignal-to-Noise or SNR) and/or spatial scalability may be implemented asfollows. For a base layer, a conventional non-scalable video encoder anddecoder is used. The reconstructed/decoded pictures of the base layerare included in the reference picture buffer for an enhancement layer.In H.264/AVC, HEVC, and similar codecs using reference picture list(s)for inter prediction, the base layer decoded pictures may be insertedinto a reference picture list(s) for coding/decoding of an enhancementlayer pictures similarly to the decoded reference pictures of theenhancement layer. Consequently, the encoder may choose a base-layerreference picture as inter prediction reference and may indicate its usee.g. with a reference picture index in the coded bitstream. The decoderdecodes from the bitstream, for example from a reference picture index,that a base-layer picture is used as an inter prediction reference forthe enhancement layer. When a decoded base-layer picture is used as aprediction reference for an enhancement layer, it is referred to as aninter-layer reference picture.

Various technologies for providing three-dimensional (3D) video contentare currently investigated and developed. Especially, intense studieshave been focused on various multiview applications wherein a viewer isable to see only one pair of stereo video from a specific viewpoint andanother pair of stereo video from a different viewpoint. One of the mostfeasible approaches for such multiview applications has turned out to besuch wherein only a limited number of views, e.g. a mono or a stereovideo plus some supplementary data, is provided to a decoder side andall required views are then rendered (i.e. synthesized) locally be thedecoder to be displayed on a display.

Some key definitions, bitstream and coding structures, and concepts ofH.264/AVC and HEVC are described in this section as an example of avideo encoder, decoder, encoding method, decoding method, and abitstream structure, wherein the embodiments may be implemented. Some ofthe key definitions, bitstream and coding structures, and concepts ofH.264/AVC are the same as in HEVC—hence, they are described belowjointly. The aspects of the invention are not limited to H.264/AVC orHEVC, but rather the description is given for one possible basis on topof which the invention may be partly or fully realized.

When describing H.264/AVC and HEVC as well as in example embodiments,common notation for arithmetic operators, logical operators, relationoperators, bit-wise operators, assignment operators, and range notatione.g. as specified in H.264/AVC or HEVC may be used. Furthermore, commonmathematical functions e.g. as specified in H.264/AVC or HEVC may beused and a common order or precedence and execution order (from left toright or from right to left) of operators e.g. a specified in H.264/AVCor HEVC may be used.

When describing H.264/AVC and HEVC as well as in example embodiments,the following description may be used to specify the parsing process ofeach syntax element.

-   -   b(8): byte having any pattern of bit string (8 bits).    -   se(v): signed integer Exp-Golomb-coded syntax element with the        left bit first.    -   u(n): unsigned integer using n bits. When n is “v” in the syntax        table, the number of bits varies in a manner dependent on the        value of other syntax elements. The paring process for this        descriptor is specified by n next bits from the bitstream        interpreted as a binary representation of an unsigned integer        with the most significant bit written first.    -   ue(v): unsigned integer Exp-Golomb-coded syntax element with the        left bit first.

An Exp-Golomb bit string may be converted to a code number (codeNum) forexample using the following table:

Bit string codeNum 1 0 0 1 0 1 0 1 1 2 0 0 1 0 0 3 0 0 1 0 1 4 0 0 1 1 05 0 0 1 1 1 6 0 0 0 1 0 0 0 7 0 0 0 1 0 0 1 8 0 0 0 1 0 1 0 9 . . . . ..

A code number corresponding to an Exp-Golomb bit string may be convertedto se(v) for example using the following table:

codeNum syntax element value 0 0 1 1 2 −1 3 2 4 −2 5 3 6 −3 . . . . . .

When describing H.264/AVC and HEVC as well as in example embodiments,syntax structures, semantics of syntax elements, and decoding processmay be specified as follows. Syntax elements in the bitstream arerepresented in bold type. Each syntax elements is described by its name(all lower case letters with underscore characteristics), optionally itsone or two syntax categories, and one or two descriptors for its methodof coded representation. The decoding process behaves according to thevalue of the syntax element and to the values of previously decodedsyntax elements. When a value of a syntax element is used in the syntaxtables or the text, named by a mixture of lower case and upper caseletter and without any underscore characters. Variables starting with anupper case letter are derived for the decoding of the current syntaxstructure and all depending syntax structures. Variables starting withan upper case letter may be used in the decoding process for latersyntax structures without mentioning the originating syntax structure ofthe variable. Variables starting with a lower case letter are only usedwithin the context in which they are derived. In some cases, “mnemonic”names for syntax element values or variable values are usedinterchangeably with their numerical values. Sometimes “mnemonic” namesare used without any associated numerical values. The association ofvalues and names is specified in the text. The names are constructedfrom one or more groups of letters separated by an underscore character.Each group starts with an upper case letter and may contain more uppercase letters.

When describing H.264/AVC and HEVC as well as in example embodiments, asyntax structure may be specified using the following. A group ofstatements enclosed in curly brackets is a compound statement and istreated functionally as a single statement. A “while” structurespecifies a test of whether a condition is true, and if true, specifiesevaluation of a statement (or compound statement) repeatedly until thecondition is no longer true. A “do . . . while” structure specifiesevaluation of a statement once, followed by a test of whether acondition is true, and if true, specifies repeated evaluation of thestatement until the condition is no longer true. An “if . . . else”structure specifies a test of whether a condition is true, and if thecondition is true, specifies evaluation of a primary statement,otherwise, specifies evaluation of an alternative statement. The “else”part of the structure and the associated alternative statement isomitted if no alternative statement evaluation is needed. A “for”structure specifies evaluation of an initial statement, followed by atest of a condition, and if the condition is true, specifies repeatedevaluation of a primary statement followed by a subsequent statementuntil the condition is no longer true.

In the description of existing standards as well as in the descriptionof example embodiments, a phrase “by external means” or “throughexternal means” may be used. For example, an entity, such as a syntaxstructure or a value of a variable used in the decoding process, may beprovided “by external means” to the decoding process. The phrase “byexternal means” may indicate that the entity is not included in thebitstream created by the encoder, but rather conveyed externally fromthe bitstream for example using a control protocol. It may alternativelyor additionally mean that the entity is not created by the encoder, butmay be created for example in the player or decoding control logic oralike that is using the decoder. The decoder may have an interface forinputting the external means, such as variable values.

Similarly to many earlier video coding standards, the bitstream syntaxand semantics as well as the decoding process for error-free bitstreamsare specified in H.264/AVC and HEVC. The encoding process is notspecified, but encoders must generate conforming bitstreams. Bitstreamand decoded conformance can be verified with the Hypothetical ReferenceDecoder losses, but the use of the tools in encoding is optional and nodecoding process has been specified for erroneous bitstreams.

The elementary unit for the input to an H.264/AVC or HEVC encoder andthe output of an H.264/AVC or HEVC decoder, respectively, is a picture.A picture may either be a frame or a field. A frame comprises a matrixof luma samples and corresponding chroma samples. A field is a set ofalternate sample rows of a frame and may be used as encoder input, whenthe source signal is interlaced. Chroma pictures may be subsampled whencompared to luma pictures. For example, in the 4:2:0 sampling patternthe spatial resolution of chroma pictures is half of that of the lumapicture along both coordinate axes.

In H.264/AVC, a macroblock is a 16×16 block of luma samples and thecorresponding blocks of chroma samples. For example, in the 4:2:0sampling pattern, a macroblock contains one 8×8 block of chroma samplesper each chroma component. In H.264/AVC, a picture is partitioned to oneor more slice groups, and a slice group contains one or more slices. InH.264/AVC, a slice consists of an integer number of macroblocks orderedconsecutively in the raster scan within a particular slice group.

In HEVC, video pictures are divided into coding units (CU) covering thearea of the picture. A CU consists of one or more prediction units (PU)defining the prediction process for the samples within the CU and one ormore transform units (TU) defining the prediction error coding processfor the samples in the CU. Typically, a CU consists of a square block ofsamples with a size selectable from a predefined set of possible CUsizes. A CU with the maximum allowed size is typically named as CTU(coding tree unit) and the video picture is divided into non-overlappingCTUs. An CTU can be further split into a combination of smaller CUs,e.g. by recursively splitting the CTU and resultant CUs. Each resultingCU typically has at least one PU and at least on TU associated with it.Each PU and TU can further be split into smaller PUs and TUs in order toincrease granularity of the prediction and prediction error codingprocesses, respectively. The PU splitting can be realized by splittingthe CU into four equal size square PUs or splitting the CU into tworectangle PUs vertically or horizontally in a symmetric or asymmetricway. The division of the image into CUs, and division of CUs into PUsand TUs is typically signaled in the bitstream allowing the decoder toreproduce the intended structure of these units

In HEVC, a picture can be partitioned in tiles, which are rectangularand contain an integer number of CTUs. In HEVC, the partitioning totiles forms a regular grid, where heights and widths of tiles differfrom each other by one CTU at the maximum.

In HEVC, a slice is defined to be an integer number of coding tree unitscontained in one independent slice segment and all subsequent dependentslice segments (if any) that precede the next independent slice segment(if any) within the same access unit. In HEVC, a slice segment isdefined to be an integer number of coding tree units orderedconsecutively in the tile scan and contained in a single NAL unit. Thedivision of each picture into slice segments is a partitioning. In HEVC,an independent slice segment is defined to be a slice segment for whichthe values of the syntax elements of the slice segment header are notinferred from the values for a preceding slice segment, and a dependentslice segment is defined to be a slice segment for which the values ofsome syntax elements of the slice segment header are inferred from thevalues for the preceding independent slice segment in decoding order. InHEVC, a slice header is defined to be the slice segment header of theindependent slice segment that is a current slice segment or is theindependent slice segment that precedes a current dependent slicesegment, and a slice segment header is defined to be a part of a codedslice segment containing the data elements pertaining to the first orall coding tree units represented in the slice segment. The CUs arescanned in the raster scan order of LCUs within tiles or within apicture, if tiles are not in use. Within an LCU, the CUs have a specificscan order.

In a Working Draft (WD) 5 of HEVC, some key definitions and concepts forpicture partitioning are defined as follows. A partitioning is definedas the division of a set into subsets such that each element of the setis in exactly one of the subsets.

A basic coding unit in a HEVC WD5 is a treeblock. A treeblock is an N×Nblock of luma samples and two corresponding blocks of chroma samples ofa picture that has three sample arrays, or an N×N block of samples of amonochrome picture or a picture that is coded using three separatecolour planes. A treeblock may be partitioned for different coding anddecoding processes. A treeblock partition is a block of luma samples andtwo corresponding blocks of chroma samples resulting from a partitioningof a treeblock for a picture that has three sample arrays or a block ofluma samples resulting from a partitioning of a treeblock for amonochrome picture or a picture that is coded using three separatecolour planes. Each treeblock is assigned a partition signalling toidentify the block sizes for intra or inter prediction and for transformcoding. The partitioning is a recursive quadtree partitioning. The rootof the quadtree is associated with the treeblock. The quadtree is splituntil a leaf is reached, which is referred to as the coding node. Thecoding node is the root node of two tress, the prediction tree and thetransform tree. The prediction tree specifies the position and size ofprediction blocks. The prediction tree and associated prediction dataare referred to as a prediction unit. The transform tree specifies theposition and size of transform blocks. The transform tree and associatedtransform data are referred to as a transform unit. The splittinginformation for luma and chroma is identical for the prediction tree andmay or may not be identical for the transform tree. The coding node andthe associated prediction and transform units form together a codingunit.

In a HEVC WD5, pictures are divided into slices and tiles. A slice maybe a sequence of treeblocks but (when referring to a so-called finegranular slice) may also have its boundary within a treeblock at alocation where a transform unit and prediction unit coincide. Treeblockswithin a slice are coded and decoded in a raster scan order. For theprimary coded picture, the division of each picture into slices is apartitioning.

In a HEVC WD5, a tile is defined as an integer number of treeblocksco-occurring in one column and one row, ordered consecutively in theraster scan within the tile. For the primary coded picture, the divisionof each picture into tiles is a partitioning. Tiles are orderedconsecutively in the raster scan within the picture. Although a slicecontains treeblocks that are consecutive in the raster scan within atile, these treeblocks are not necessarily consecutive in the rasterscan within the picture. Slices and tiles need not contain the samesequence of treeblocks. A tile may comprise treeblocks contained in morethan one slice. Similarly, a slice may comprises treeblocks contained inseveral tiles.

In H.264/AVC and HEVC, in-picture prediction may be disabled acrossslice boundaries. Thus, slices can be regarded as a way to split a codedpicture into independently decodable pieces, and slices are therefore ofthe regarded as elementary units for transmission. In many cases,encoders may indicate in the bitstream which types of in-pictureprediction are turned off across slice boundaries, and the decoderoperation takes this information into account for example whenconcluding which prediction sources are available. For example, samplesfrom a neighboring macroblock or CU may be regarded as unavailable forintra prediction, if the neighboring macroblock or CU resides in adifferent slice.

A syntax element may be defined as an element of data represented in thebitstream. A syntax structure may be defined as zero or more syntaxelements present together in the bitstream in a specified order.

The elementary unit for the output of an H.264/AVC or HEVC encoder andthe input of an H.264/AVC or HEVC decoder respectively, is a NetworkAbstraction Layer (NAL) unit. For transport over packet-orientednetworks or storage into structured files, NAL units may be encapsulatedinto packets or similar structures. A bytestream format has beenspecified in H.264/AVC and HEVC for transmission or storage environmentsthat do not provide framing structures. The bytestream format separatesNAL units from each other by attaching a start code in front of each NALunit. To avoid false detection of NAL unit boundaries, encoders may runa byte-oriented start code emulation prevention algorithm, which adds anemulation prevention byte to the NAL unit payload if a start code wouldhave occurred otherwise. In order to enable straightforward gatewayoperation between packet- and stream-oriented systems, start codeemulation prevention may always be performed regardless of whether thebytestream format is in use or not.

NAL units consist of a header and payload. In H.264/AVC, the NAL unitheader indicates the type of the NAL unit and whether a coded slicecontained in the NAL unit is a part of a reference picture or anon-reference picture. H.264/AVC includes a 2-bit nal_ref_idc syntaxelement, which when equal to 0 indicates that a coded slice contained inthe NAL unit is a part of a non-reference picture and when greater than0 indicates that a coded slice contained in the NAL unit is a part of areference picture. The header for SVC and MVC NAL units may beadditionally contain various indications related to scalability andmultiview hierarchy.

In HEVC, a two-byte NAL unit header is used for all specified NAL unittypes. The NAL unit header contains one reserved bit, a six-bit NAL unittype indication, a six-bit reserved field (called nuh_layer_id) and athree-bit temporal_id_plus1 indication for temporal level. Thetemporal_id_plus1 syntax element may be regarded as a temporalidentifier for the NAL unit, and a zero-based TemporalId variable may bederived as follows: TemporalId=temporal_id_plus1−1. TemporalId equal to0 corresponds to the lowest temporal level. The value oftemporal_id_plus1 is required to be non-zero in order to avoid startcode emulation involving the two NAL unit header bytes. The bitstreamcreated by excluding all VCL NAL units having a TemporalId greater thanor equal to a selected value and including all other VCL NAL unitsremains conforming. Consequently, a picture having TemporalId equal toTID does not use any picture having a TemporalId greater than TID asinter prediction reference. A sub-layer or a temporal sub-layer may bedefined to be a temporal scalable layer of a temporal scalablebitstream, consisting of VCL NAL units with a particular value of theTemporalId variable and the associated non-VCL NAL units. Without lossof generality, in some example embodiments a variable LayerId is derivedfrom the value of nuh_layer_id for example as follows:LayerId=nuh_layer_id. In the following, LayerId, nuh_layer_id andlayer_id are used interchangeably unless otherwise indicated.

It is expected that nuh_layer_id and/or similar syntax elements in NALunit header would carry information on the scalability hierarchy. Forexample, the LayerId value may be mapped to values of variables orsyntax elements describing different scalability dimensions, such asquality_id or similar, dependency_id or similar, any other type of layeridentifier, view order index or similar, view identifier, an indicationwhether the NAL unit concerns depth or texture i.e. depth_flag orsimilar, or an identifier similar to priority_id of SVC indicating avalid sub-bitstream extraction if all NAL units greater than a specificidentifier value are removed from the bitstream. nuh_layer_id and/orsimilar syntax elements may be partitioned into one or more syntaxelements indicating scalability properties. For example, a certainnumber of bits among nuh_layer_id and/or similar syntax elements may beused for dependency_id or similar, while another certain number of bitsamong nuh_layer_id and/or similar syntax elements may be used forquality_id or similar. Alternatively, a mapping of LayerId values orsimilar to values of variables or syntax elements describing differentscalability dimensions may be provided for example in a Video ParameterSet, a Sequence Parameter Set or another syntax structure.

NAL units can be categorized into Video Coding Layer (VCL) NAL units andnon-VCL NAL units. VCL NAL units are typically coded slice NAL units. InH.264/AVC, coded slice NAL units contain syntax elements representingone or more coded macroblocks, each of which corresponds to a block ofsamples in the uncompressed picture. In HEVC, coded slice NAL unitscontain syntax elements representing one or more CU.

In H.264/AVC, a coded slice NAL unit can be indicated to be a codedslice in an Instantaneous Decoding Refresh (IDR) picture or coded slicein a non-IDR picture.

In HEVC, a coded slice NAL unit can be indicated to be one of thefollowing types:

Name of Content of NAL unit and RBSP nal_unit_type nal_unit_type syntaxstructure 0, 1 TRAIL_N, Coded slice segment of a non-TSA, TRAIL_Rnon-STSA trailing picture slice_segment_layer_rbsp( ) 2, 3 TSA_N, Codedslice segment of a TSA picture TSA_R slice_segment_layer_rbsp( ) 4, 5STSA_N, Coded slice segment of an STSA STSA_R picture slice_layer_rbsp() 6, 7 RADL_N, Coded slice segment of a RADL RADL_R pictureslice_layer_rbsp( ) 8, 9 RASL_N, Coded slice segment of a RASL RASL_R,picture slice_layer_rbsp( ) 10, 12, 14 RSV_VCL_N10 Reserved // reservednon-RAP non- RSV_VCL_N12 reference VCL NAL unit types RSV_VCL_N14 11,13, 15 RSV_VCL_R11 Reserved // reserved non-RAP RSV_VCL_R13 referenceVCL NAL unit types RSV_VCL_R15 16, 17, 18 BLA_W_LP Coded slice segmentof a BLA picture BLA_W_DLP (a.ka. slice_segment_layer_rbsp( )BLA_W_RADL) BLA_N_LP 19, 20 IDR_W_DLP (a.k.a Coded slice segment of anIDR picture IDR_W_RADL) slice_segment_layer_rbsp( ) IDR_N_LP 21 CRA_NUTCoded slice segment of a CRA picture slice_segment_layer_rbsp( ) 22, 23RSV_RAP_VCL22 . . . RSV_RAP_VCL23 Reserved // reserved RAP VCL NAL unittypes 24 . . . 31 RSV_VCL24 . . . RSV_VCL31 Reserved // reserved non-RAPVCL NAL unit types

In a draft HEVC standard, abbreviations for picture types may be definedas follows: trailing (TRAIL) picture, Temporal Sub-layer Access (TSA),Step-wise Temporal Sub-layer Access (STSA), Random Access DecodableLeading (RADL) picture, Random Access Skipped Leading (RASL) picture,Broken Link Access (BLA) picture, Instantaneous Decoding Refresh (IDR)picture, Clean Random Access (CRA) picture.

A Random Access Point (RAP) picture, which may also or alternatively bereferred to as intra random access point (IRAP) picture, is a picturewhere each slice or slice segment has nal_unit_type in the range of 16to 23, inclusive. A RAP picture contains only intra-coded slices, andmay be a BLA picture, a CRA picture or an IDR picture. The first picturein the bitstream is a RAP picture. Provided the necessary parameter setsare available when they need to be activated, the RAP picture and allsubsequent non-RASL pictures in decoding order can be correctly decodedwithout performing the decoding process of any pictures that precede theRAP picture in decoding order. There may be pictures in a bitstream thatcontain only intra-coded slices that are not RAP pictures.

In HEVC a CRA picture may be the first picture in the bitstream indecoding order, or may appear later in the bitstream. CRA pictures inHEVC allow so-called leading pictures that follow the CRA picture indecoding order but precede it in output order. Some of the leadingpictures, so-called RASL pictures, may use pictures decoded before theCRA picture as a reference. Pictures that follow a CRA picture in bothdecoding and output order are decodable if random access is performed atthe CRA picture, and hence clean random access is achieved similarly tothe clean random access functionality of an IDR picture.

A CRA picture may have associated RADL or RASL pictures. When a CRApicture is the first picture in the bitstream in decoding order, the CRApicture is the first picture of a coded video sequence in decodingorder, and any associated RASL pictures are not output by the decoderand may not be decodable, as they may contain references to picturesthat are not present in the bitstream.

A leading picture is a picture that precedes the associated RAP picturein output order. The associated RAP picture is the previous RAP picturein decoding order (if present). A leading picture may either be a RADLpicture or a RASL picture.

All RASL pictures are leading pictures of an associated BLA or CRApicture. When the associated RAP picture is a BLA picture or is thefirst coded picture in the bitstream, the RASL picture is not output andmay not be correctly decodable, as the RASL picture may containreferences to pictures that are not present in the bitstream. However, aRASL picture can be correctly decoded if the decoding had started from aRAP picture before the associated RAP picture of the RASL picture. RASLpictures are not used as reference pictures for the decoding process ofnon-RASL pictures. When present, all RASL pictures precede, in decodingorder, all trailing pictures of the same associated RAP picture. In somedrafts of the HEVC standard, a RASL picture was referred to a Tagged forDiscard (TFD) picture.

All RADL pictures are leading pictures. RADL pictures are not used asreference pictures for the decoding process of trailing pictures of thesame associated RAP picture. When present, all RADL pictures precede, indecoding order, all trailing pictures of the same associated RAPpicture. RADL pictures do not refer to any picture preceding theassociated RAP picture in decoding order and can therefore be correctlydecoded when the decoding starts from the associated RAP picture. Insome drafts of the HEVC standard, a RADL picture was referred to aDecodable Leading Picture (DLP).

Decodable leading pictures may be such that can be correctly decodedwhen the decoding is started from the CRA picture. In other words,decodable leading pictures use only the initial CRA picture orsubsequent pictures in decoding order as reference in inter prediction.Non-decodable leading pictures are such that cannot be correctly decodedwhen the decoding is started from the initial CRA picture. In otherwords, non-decodable leading pictures use pictures prior, in decodingorder, to the initial CRA picture as references in inter prediction.

Concatenation of coded video data, which may also be referred to assplicing, may occur for example coded video sequences are concatenatedinto a bitstream that is broadcast or streamed or stored in a massmemory. For example, coded video sequences representing commercials oradvertisements may be concatenated with movies or other “primary”content. A spliced bitstream may be defined as the bitstream that isadded after a first bitstream and the concatenation of the firstbitstream and the spliced bitstream form a new bitstream. If a splicedbitstream contains only one coded video sequence, it may be referred toas the spliced coded video sequence. Similarly, the first coded videosequence of the spliced bitstream may be referred to as the splicedcoded video sequence. An entity performing the concatenation of thebitstreams may be referred to as a splicer.

When a part of a bitstream starting from a CRA picture is included inanother bitstream, the RASL pictures associated with the CRA picturemight not be correctly decodable, because some of their referencepictures might not be present in the combined bitstream. To make such asplicing operation straightforward, the NAL unit type of the CRA picturecan be changed to indicate that it is a BLA picture. The RASL picturesassociated with a BLA picture may not be correctly decodable hence arenot be output/displayed. Furthermore, the RASL pictures associated witha BLA picture may be omitted from decoding.

A BLA picture may be the first picture in the bitstream in decodingorder, or may appear later in the bitstream. Each BLA picture begins anew coded video sequence, and has similar effect on the decoding processas an IDR picture. However, a BLA picture contains syntax elements thatspecify a non-empty reference picture set. When a BLA picture hasnal_unit_type equal to BLA_W_LP, it may have associated RASL pictures,which are not output by the decoder and may not be decodable, as theymay contain references to pictures that are not present in thebitstream. When a BLA picture has nal_unit_type equal to BLA_W_LP, itmay also have associated RADL pictures, which are specified to bedecoded. When a BLA picture has nal_unit_type equal to BLA_W_DLP, itdoes not have associated RASL pictures but may have associated RADLpictures, which are specified to be decoded. BLA_W_DLP may also bereferred to as BLA_W_RADL. When a BLA picture has nal_unit_type equal toBLA_N_LP, it does not have any associated leading pictures.

An IDR picture having nal_unit_type equal to IDR_N_LP does not haveassociated leading pictures present in the bitstream. An IDR picturehaving nal_unit_type equal to IDR_W_DLP does not have associated RASLpictures present in the bitstream, but may have associated RADL picturesin the bitstream. IDR_W_DLP may also be referred to as IDR_W_RADL.

When the value of nal_unit_type is equal to TRAIL_N, TSA_N, STSA_N,RADL_N, RASL_N, RSV_VCL_N10, RSV_VCL_N12, or RSV_VCL_N14, the decodedpicture is not used as a reference for any other picture of the sametemporal sub-layer and the picture may be referred to as a sub-layernon-reference picture. That is, in HEVC, when the value of nal_unit_typeis equal to TRAIL_N, TSA_N, STSA_N, RADL_N, RASL_N, RSV_VCL_N10,RSV_VCL_N12, or RSV_VCL_N14, the decoded picture is not included in anyof RefPicSetStCurrBefore, RefPicSetStCurrAfter and RefPicSetLtCurr ofany picture with the same value of TemporalId. A coded picture withnal_unit_type equal to TRAIL_N, TSA_N, STSA_N, RADL_N, RASL_N,RSV_VCL_N10, RSV_VCL_N12, or RSV_VCL_N14 may be discarded withoutaffecting the decodability of other pictures with the same value ofTemporalId. Pictures that are not sub-layer non-reference pictures maybe referred to as sub-layer reference pictures. A sub-layer referencepicture may be defined as a picture that contains samples that may beused for inter prediction in the decoding process of subsequent picturesof the same sub-layer in decoding order.

A trailing picture may be defined as a picture that follows theassociated RAP picture in output order. Any picture that is a trailingpicture does not have nal_unit_type equal to RADL_N, RADL_R, RASL_N orRASL_R. Any picture that is a leading picture may be constrained toprecede, in decoding order, all trailing pictures that are associatedwith the same RAP picture. No RASL pictures are present in the bitstreamthat are associated with a BLA picture having nal_unit_type equal toBLA_W_DLP or BLA_N_LP. No RADL pictures are present in the bitstreamthat are associated with a BLA picture having nal_unit_type equal toBLA_N_LP or that are associated with an IDR picture having nal_unit_typeequal to IDR_N_LP. Any RASL picture associated with a CRA or BLA picturemay be constrained to precede any RADL picture associated with the CRAor BLA picture in output order. Any RASL picture associated with a CRApicture may be constrained to follow, in output order, any other RAPpicture that precedes the CRA picture in decoding order.

In HEVC, there are two picture types, the TSA and STSA picture types,that can be used to indicate temporal sub-layer switching points. Iftemporal sub-layers with TemporalId up to N had been decoded until theTSA or STSA picture (exclusive) and the TSA or STSA picture hasTemporalId equal to N+1, the TSA or STSA picture enables decoding of allsubsequent pictures (in decoding order) having TemporalId equal to N+1.The TSA picture type may impose restrictions on the TSA picture itselfand all pictures in the same sub-layer that follow the TSA picture indecoding order. None of these pictures is allowed to use interprediction from any picture in the same sub-layer that precedes the TSApicture in decoding order. The TSA definition may further imposerestrictions on the pictures in higher sub-layers that follow the TSApicture in decoding order. None of these pictures is allowed to refer apicture that precedes the TSA picture in decoding order if that picturebelongs to the same or higher sub-layer as the TSA picture. TSA pictureshave TemporalId greater than 0. The STSA is similar to the TSA picturebut does not impose restrictions on the pictures in higher sub-layersthat follow the STSA picture in decoding order and hence enableup-switching only onto the sub-layer where the STSA picture resides.

A non-VCL NAL unit may be for example one of the following types: asequence parameter set, a picture parameter set, a supplementalenhancement information (SEI) NAL unit, an access unit delimiter, an endof sequence NAL unit, an end of stream NAL unit, or a filler data NALunit. Parameter sets may be needed for the reconstruction of decodedpictures, whereas many of the other non-VCL NAL units are not necessaryfor the reconstruction of decoded sample values.

Parameters that remain unchanged through a coded video sequence may beincluded in a sequence parameter set (SPS). In addition to theparameters that may be essential to the decoding process, the sequenceparameter set may optionally contain video usability information (VUI),which includes parameters that may be important for buffering, pictureoutput timing, rendering and resource reservation. There are three NALunits specified in H.264/AVC to carry sequence parameter sets: thesequence parameter set NAL unit containing all the data for H.264/AVCVCL NAL units in the sequence, the sequence parameter set extension NALunit containing the data for auxiliary coded pictures, and the subsetsequence parameter set for MVC and SVC VCL NAL units. A pictureparameter set (PPS) contains such parameters that are likely to beunchanged in several coded pictures.

Parameter set syntax structures may have extensions mechanisms, whichmay for example be used to include parameters that are specific toextensions of a coding standard. An example syntax of an extensionmechanism is provided in the following for SPS:

seq_parameter_set_rbsp( ) { Descriptor  ...  sps_extension_flag u(1) if( sps_extension_flag )   while( more_rbsp_data( ) )   sps_extension_data_flag u(1)  rbsp_trailing_bits( ) }

Decoders of particular version(s) of a coding standard or a codingscheme may ignore sps_extension_data_flag, while in another version ofthe coding standard or the coding scheme, an extension syntax structuremay be specified and may appear within the sps_extension_data_flag bits.Similar extensions mechanisms may be specifies also for other types ofparameter sets.

In a draft version of HEVC, there is also a third type of parametersets, here referred to as Adaptation Parameter Set (APS), which includesparameters that are likely to be unchanged in several coded slices. In adraft version of HEVC, the APS syntax structure includes parameters orsyntax elements related to context-based adaptive binary arithmeticcoding (CABAC), adaptive sample offset, adaptive loop filtering, anddeblocking filtering. In a draft version of HEVC, an APS is a NAL unitand coded without reference or prediction from any other NAL unit. Anidentifier, referred to as aps_id syntax element, is included in APS NALunit, and included and used in the slice header to refer to a particularAPS. However, APS was not included in the final H.265/HEVC standard.

H.265/HEVC also includes another type of a parameter set, called a videoparameter set (VPS). A video parameter set RBSP may include parametersthat can be referred to by one or more sequence parameter set RBSPs.

The relationship and hierarchy between VPS, SPS, and PPS may bedescribed as follows. VPS resides one level above SPS in the parameterset hierarchy and in the context of scalability and/or 3DV. VPS mayinclude parameters that are common for all slices across all(scalability or view) layers in the entire coded video sequence. SPSincludes the parameters that are common for all slices in a particular(scalability or view) layer in the entire coded video sequence, and maybe shared by multiple (scalability or view) layers. PPS includes theparameters that are common for all slices in a particular layerrepresentation (the representation of one scalability or view layer inone access unit) and are likely to be shared by all slices in multiplelayer representations.

VPS may provide information about the dependency relationships of thelayers in a bitstream, as well as many other information that areapplicable to all slices across all (scalability or view) layers in theentire coded video sequence. In a scalable extension of HEVC, VPS mayfor example include a mapping of the LayerId value derived from the NALunit header to one or more scalability dimension values, for examplecorrespond to dependency_id, quality_id, view_id, and depth_flag for thelayer defined similarly to SVC and MVC. VPS may include profile andlevel information for one or more layers as well as the profile and/orlevel for one or more temporal sub-layers (consisting of VCL NAL unitsat and below certain TemporalId values) of a layer representation. VPSmay also provide the maximum number of layers present in the bitstream.For example, the syntax element vps_max_layers_minus1 may be included inthe syntax and vps_max_layer_minus1+1 may indicate the maximum number oflayers present in the bitstream. The actual number of layers in thebitstream may be smaller than or equal to vps_max_layers_minus1+1. Thevariable MaxLayersMinus1 may be specified to be equal tovps_max_layers_minus1 and may indicate the maximum number of layersminus 1 in the CVS for which the VPS is the active VPS.

An example syntax of a VPS extension intended to be a part of the VPS isprovided in the following. The presented VPS extension provides thedependency relationships among other things. It should be understoodthat the VPS extension syntax is provided as an example and othersimilar and/or extended syntax structures may be equivalently appliedwith different embodiments.

vps_extension( ) { Descriptor  while( !byte_aligned( ) )  vps_extension_byte_alignment_reserved_one_bit u(1)  for( i = 0,NumScalabilityTypes = 0; i < 16; i++ ) {   scalability_mask[ i ] u(1)  NumScalabilityTypes += scalability_mask[ i ]  }  for( j = 0; j <NumScalabilityTypes; j++ )   dimension_id_len_minus1[ j ] u(3) vps_nuh_layer_id_present_flag u(1)  for( i = 0; i <=vps_max_layers_minus1; i++ ) {   if( vps_nuh_layer_id_present_flag &&i > 0 )    layer_id_in_nuh[ i ] u(6)   for( j = 0; j <NumScalabilityTypes; j++ )    dimension_id[ i ][ j ] u(v)  }  for( i =1; i <= vps_max_layers_minus1; i++ )   for(j = 0; j < i; j++ )   direct_dependency_flag[ i ][ j ] u(1)  direct_dep_type_len_minus2ue(v)  for( i = 1; i <=vps_max_layers_minus1; i++ )   for( j = 0; j < i;j++ )    if( direct_dependency_flag[ i ][ j ] )    direct_dependency_type[ i ][ j ] u(v) }

The semantics of the presented VPS extension may be specified asdescribed in the following paragraphs.

vps_extension_byte_alignment_reserved_one_bit is equal to 1 and is usedto achieve alignment of the next syntax element to a byte boundary.

scalability_mask[i] equal to 1 indicates that dimension_id syntaxelements corresponding to the i-th scalability dimension are present.scalability_mask[i] equal to 0 indicates that dimension_id syntaxelements corresponding to the i-th scalability dimension are notpresent. The scalability dimensions corresponding to each value of of iin scalability_mask[i] may be specified for example to include thefollowing or any subset thereof along with other scalability dimensions.

scalability_mask Scalability ScalabilityId index dimension mapping 0multiview ViewId 1 spatial or quality DependencyId scalability

dimension_id_len_minus1[j] plus1 specifies the length, in bits, of thedimension_id[i][j] syntax element. vps_nuh_layer_id_present_flagspecifies whether the layer_id_in_nuh[i] syntax is present.layer_id_in_nuh[i] specifies the value of the nuh_layer_id syntaxelement in VCL NAL units of the i-th layer. When not present, the valueof layer_id_in_nuh[i] is inferred to be equal to i. layer_id_in_nuh[i]is greater than layer_id_in_nuh[i−1]. The variableLayerIdxInVps[layer_id_in_nuh[i]] is set equal to i. dimension_id[i][j]specifies the identifier of the j-th scalability dimension type of thei-th layer. When not present, the value of dimension_id[i][j] isinferred to be equal to 0. The number of bits used for therepresentation of dimension_id[i][j] is dimension_id_len_minus1[j]+1bits.

direct_dependency_flag[i][j] equal to 0 specifies that the layer withindex j is not a direct reference layer for the layer with index i.direct_dependency_flag[i][j] equal to 1 specifies that the layer withindex j may be a direct reference layer for the layer with index i. Whendirect_dependency_flag[i][j] is not present for i and j in the range of0 to vps_max_num_layers_minus1, it is inferred to be equal to 0.

The variables NumDirectRefLayers[i] and RefLayerId[i][j] may be derivedas follows:

for( i = 1; i <= vps_max_layers_minus1; i++ ) for( j = 0,NumDirectRefLayers[ i ] = 0; j < i; j++ ) if( direct_dependency_flag[ i][ j ] == 1 ) RefLayerId[ i ][ NumDirectRefLayers[ i ]++ ] =layer_id_in_nuh[ j ]

direct_dep_type_len_minus2 plus2 specifies the number of bits of thedirect_dependency_type[i][j] syntax element.direct_dependency_type[i][j] equal to 0 indicates that sample predictionmay be used and motion prediction is not used for layer identified by ifrom layer identified by j. direct_dependency_type[i][j] equal to 1indicates that motion prediction may be used and sample prediction isnot used for layer identified by i from layer identified by j.direct_dependency_type[i][j] equal to 2 indicates that both sample andmotion prediction may be used for layer identified by i from layeridentified by j.

The variables NumSamplePredRefLayers[i], NumMotionPredRefLayers[i],SamplePredEnabledFlag[i][j], MotionPredEnabledFlag[i][j],NumDirectRefLayers[i], RefLayerId[i][j], MotionPredRefLayerId[i][j], andSamplePredRefLayerId[i][j] may be derived as follows:

for( i = 0; i < 64; i++ ) { NumSamplePredRefLayers[ i ] = 0NumMotionPredRefLayers[ i ] = 0 NumDirectRefLayers[ i ] = 0 for( j = 0;j < 64; j++ ) {  SamplePredEnabledFlag[ i ][ j ] = 0 MotionPredEnabledFlag[ i ][ j ] = 0  RefLayerId[ i ][ j ] = 0 SamplePredRefLayerId[ i ][ j ] = 0  MotionPredRefLayerId[ i ][ j ] = 0} } for( i = 1; i <= vps_max_layers_minus1; i++ ) {  iNuhLId =layer_id_in_nuh[ i ]  for( j = 0; j < i; j++ ) if(direct_dependency_flag[ i ] [ j ] ) { RefLayerId[ iNuhLId ][NumDirectRefLayers[ iNuhLId ]++ ] = layer_id_in_nuh[ j ]SamplePredEnabledFlag[ iNuhLId ][ j ] = ( ( direct_dependency_type[ i ][j ] + 1 ) & 1 )  NumSamplePredRefLayers[ iNuhLId ] += SamplePredEnabledFlag[ iNuhLId ][ j ]  MotionPredEnabledFlag[ iNuhLId][ j ] =  ( ( ( direct_dependency_type[ i ][ j ] + 1 ) & 2 ) >> 1 ) NumMotionPredRefLayers[ iNuhLId ] +=  MotionPredEnabledFlag[ iNuhLId ][j ] } } for( i = 1, mIdx = 0, sIdx = 0; i <= vps_max_layers_minus1; i++) { iNuhLId = layer_id_in_nuh[ i ] for( j = 0, j < i; j++ ) { if(MotionPredEnabledFlag[ iNuhLId ][ j ] )  MotionPredRefLayerId[ iNuhLId][ mIdx++ ] =  layer_id_in_nuh[ j ] if( SamplePredEnabledFlag[ INuhLid][ j ] )  SamplePredRefLayerId[ iNuhLid ][ sIdx++ ] =  layer_id_in_nuh[j ] } }

H.264/AVC and HEVC syntax allows many instances of parameter sets, andeach instance is identified with a unique identifier. In H.264/AVC, eachslice header includes the identifier of the picture parameter set thatis active for the decoding of the picture that contains the slice, andeach picture parameter set contains the identifier of the activesequence parameter set. Consequently, the transmission of picture andsequence parameter set does not have to be accurately synchronized withthe transmission of slices. Instead, it is sufficient that the activesequence and picture parameter sets are received at any moment beforethey are referenced, which allows transmission of parameter sets“out-of-band” using a more reliable transmission mechanism compared tothe protocols used for the slice data. For example, parameter sets canbe included as a parameter in the session description for Real-timeTransport Protocol (RTP) sessions. If parameter sets are transmittedin-band, they can be repeated to improve error robustness.

A SEI NAL unit may contain one or more SEI message, which are notrequired for the decoding of output pictures but assist in relatedprocesses, such as picture output timing, rendering, error detection,error concealment, and resource reservation. Several SEI messages arespecified in H.264/AVC and HEVC, and the user data SEI messages enableorganizations and companies to specify SEI messages for their own use.H.264/AVC and HEVC contain the syntax and semantics for the specifiedSEI messages but no process for handling the messages in the recipientis defined. Consequently, encoders are required to follow the H.264/AVCstandard or the HEVC standard when they create SEI messages, anddecoders conforming to the H.264/AVC standard or the HEVC standard,respectively, are not required to process SEI messages for output orderconformance. One of the reasons to include the syntax and semantics ofSEI messages in H.264/AVC and HEVC is to allow different systemspecifications to interpret the supplemental information identically andhence interoperate. It is intended that system specifications canrequire the use of particular SEI messages both in the encoding end andin the decoding end, and additionally the process for handlingparticular SEI messages in the recipient can be specified.

Several nesting SEI messages have been specified in the AVC and HEVCstandards or proposed otherwise. The idea of nesting SEI messages is tocontain one or more SEI messages within a nesting SEI message andprovide a mechanism for associating the contained SEI messages with asubsets of the bitstream and/or a subset of decoded data. It may berequired that a nesting SEI message contains one or more SEI messagesthat are not nesting SEI messages themselves. An SEI message containedin a nesting SEI message may be referred to as a nested SEI message. AnSEI message not contained in a nesting SEI message may be referred to asa non-nested SEI message. The scalable nesting SEI message of HEVCenables to identify either a bitstream subset (resulting from asub-bitstream extraction process) or a set of layers to which the nestedSEI messages apply. A bitstream subset may also be referred to as asub-bitstream.

A coded picture is a coded representation of a picture. A coded picturein H.264/AVC comprises the VCL NAL units that are required for thedecoding of the picture. In H.264/AVC, a coded picture can be a primarycoded picture or a redundant coded picture. A primary coded picture isused in the decoding process of valid bitstreams, whereas a redundantcoded picture is a redundant representation that should only be decodedwhen the primary coded picture cannot be successfully decoded. In HEVC,no redundant coded picture has been specified.

In H.264/AVC and HEVC, an access unit comprises a primary coded pictureand those NAL units that are associated with it. In HEVC, an access unitis defined as a set of NAL units that are associated with each otheraccording to a specified classification rule, are consecutive indecoding order, and contain exactly one coded picture. In H.264/AVC, theappearance order of NAL units within an access unit is constrained asfollows. An optional access unit delimiter NAL unit may indicate thestart of an access unit. It is followed by zero or more SEI NAL units.The coded slices of the primary coded picture appear next. In H.264/AVC,the coded slice of the primary coded picture may be followed by codedslices for zero or more redundant coded pictures. A redundant codedpicture is a coded representation of a picture or a part of a picture. Aredundant coded picture may be decoded if the primary coded picture isnot received by the decoder for example due to the loss in transmissionor a corruption in physical storage medium.

In H.264/AVC, an access unit may also include an auxiliary codedpicture, which is a picture that supplements the primary coded pictureand may be used for example in the display process. An auxiliary codedpicture may for example be used as an alpha channel or alpha planespecifying the transparency level of the samples in the decodedpictures. An alpha channel or plane may be used in a layered compositionor rendering system, where the output picture is formed by overlayingpictures being at least partly transparent on top of each other. Anauxiliary coded picture has the same syntactic and semantic restrictionsas a monochrome redundant coded picture. In H.264/AVC, an auxiliarycoded picture contains the same number of macroblocks as the primarycoded picture.

In HEVC, an access unit may be defined as a set of NAL units that areassociated with each other according to a specified classification rule,are consecutive in decoding order, and contain exactly one codedpicture. In addition to containing the VCL NAL units of the codedpicture, an access unit may also contain non-VCL-NAL units. In HEVC, thedecoding of an access unit results in a decoded picture.

In H.264/AVC, a coded video sequence is defined to be a sequence ofconsecutive access units in decoding order from an IDR access unit,inclusive, to the next IDR access unit, exclusive, or to the end of thebitstream, whichever appears earlier. In HEVC, a coded video sequence isdefined to be a sequence of access units that consists, in decodingorder, of a CRA (Clean Random Access) access unit that is the firstaccess unit in the bitstream, and IDR access unit or a BLA (Broken LinkAccess) access unit, followed by zero or more non-IDR and non-BLA accessunits including all subsequent access units up to but no including anysubsequent IDR or BLA access unit.

A group of pictures (GOP) and its characteristics may be defined asfollows. A GOP can be decoded regardless of whether any previouspictures were decoded. An open GOP is such a group of pictures in whichpictures preceding the initial intra picture in output order might notbe correctly decodable when the decoding starts from the initial intrapicture of the open GOP. In other words, pictures of an open GOP mayrefer (in inter prediction) to pictures belonging to a previous GOP. AnH.264/AVC decoder can recognize an intra picture starting an open GOPfrom the recovery point SEI message in an H.264/AVC bitstream. An HEVCdecoder can recognize an intra picture starting an open GOP, because aspecific NAL unit type, CDR NAL unit type, is used for its coded slices.A closed GOP is such a group of pictures in which all pictures can becorrectly decoded when the decoding starts from the initial intrapicture of the closed GOP. In other words, no picture in a closed GOPrefers to any pictures in previous GOPs. In H.264/AVC and HEVC, a closedGOP starts from an IDR access unit. In HEVC a closed GOP may also startfrom BLA W DLP or a BLA N LP picture. As a result, closed GOP structurehas more error resilience potential in comparison to the open GOPstructure, however at the cost of possible reduction in the compressionefficiency. Open GOP coding structure is potentially more efficient inthe compression, due to a larger flexibility in selection of referencepictures.

The bitstream syntax of H.264/AVC and HEVC indicates whether aparticular picture is a reference picture for inter prediction of anyother picture. Pictures of any coding type (I, P, B) can be referencepictures or non-reference pictures in H.264/AVC and HEVC. The NAL unitheader indicates the type of the NAL unit and whether a coded slicecontained in the NAL unit is a part of a reference picture or anon-reference picture.

Many hybrid video codecs, including H.264/AVC and HEVC, encode videoinformation in two phases. In the first phase, predictive coding isapplied for example as so-called sample prediction and/or so-calledsyntax prediction.

In the sample prediction, pixel or sample values in a certain picturearea or “block” are predicted. These pixel or sample values can bepredicted, for example, using one or more of the following ways:

-   -   Motion compensation mechanisms (which may also be referred to as        temporal prediction or motion-compensated temporal prediction or        motion-compensated prediction or MCP), which involve finding and        indicating an area in one of the previously encoded video frames        that corresponds closely to the block being coded.    -   Inter-view prediction, which involves finding and indicating an        area in one of the previously encoded view components that        corresponds closely to the block being coded.    -   View synthesis prediction, which involves synthesizing a        prediction block or image area where a prediction block is        derived on the basis of reconstructed/decoded ranging        information.    -   Inter-layer prediction using reconstructed/decoded samples, such        as the so-called IntraBL (base layer) mode of SVC.    -   Inter-layer residual prediction, in which for example the coded        residual of a reference layer or a derived residual from a        difference of a reconstructed/decoded reference layer picture        and a corresponding reconstructed/decoded enhancement layer        picture may be used for predicting a residual block of the        current enhancement layer block. A residual block may be added        for example to a motion-compensated prediction block to obtain a        final prediction block for the current enhancement layer block.    -   Intra prediction, where pixel or sample values can be predicted        by spatial mechanisms which involve finding and indicating a        spatial region relationship.

In the syntax prediction, which may also be referred to as parameterprediction, syntax elements and/or syntax element values and/orvariables derived from syntax elements are predicted from syntaxelements (de)coded earlier and/or variables derived earlier.Non-limiting examples of syntax prediction are provided below:

-   -   In motion vector prediction, motion vectors e.g. for inter        and/or inter-view prediction may be coded differentially with        respect to a block-specific predicted motion vector. In many        video codecs, the predicted motion vectors are created in a        predefined way, for example by calculating the median of the        encoded or decoded motion vectors of the adjacent blocks.        Another way to create motion vector predictions, sometimes        referred to as advanced motion vector prediction (AMVP), is to        generate a list of candidate predictions from adjacent blocks        and/or co-located blocks in temporal reference pictures and        signalling the chosen candidate as the motion vector predictor.        In addition to predicting the motion vector values, the        reference index of previously coded/decoded picture can be        predicted. The reference index is typically predicted from        adjacent blocks and/or co-located blocks in temporal reference        picture. Differential coding of motion vectors is typically        disabled across slice boundaries.    -   The block partitioning, e.g. from CTU to CUs and down to PUs,        may be predicted.    -   In filter parameter prediction, the filtering parameters e.g.        for sample adaptive offset may be predicted.

Prediction approaches using image information from a previously codedimage can also be called as inter prediction methods which may also bereferred to as temporal prediction and motion compensation. Predictionapproaches using image information within the same image can also becalled as intra prediction methods.

The second phase is one of coding the error between the predicted blockof pixels or samples and the original block of pixels or samples. Thismay be accomplished by transforming the difference in pixel or samplevalues using a specified transform. This transform may be e.g. aDiscrete Cosine Transform (DCT) or a variant thereof. After transformingthe difference, the transformed difference is quantized and entropycoded.

By varying the fidelity of the quantization process, the encoder cancontrol the balance between the accuracy of the pixel or samplerepresentation (i.e. the visual quality of the picture) and the size ofthe resulting encoded video representation (i.e. the file size ortransmission bit rate).

The decoder reconstructs the output video by applying a predictionmechanism similar to that used by the encoder in order to form apredicted representation of the pixel or sample blocks (using the motionor spatial information created by the encoder and included in thecompressed representation of the image) and prediction error decoding(the inverse operation of the prediction error coding to recover thequantized prediction error signal in the spatial domain).

After applying pixel or sample prediction and error decoding processesthe decoder combines the prediction and the prediction error signals(the pixel or sample values) to form the output video frame.

The decoder (and encoder) may also apply additional filtering processesin order to improve the quality of the output video before passing itfor display and/or storing as a prediction reference for the forthcomingpictures in the video sequence.

In many video codecs, including H.264/AVC and HEVC, motion informationis indicated by motion vectors associated with each motion compensatedimage block. Each of these motion vectors represents the displacement ofthe image block in the picture to be coded (in the encoder) or decoded(at the decoder) and the prediction source block in one of thepreviously coded or decoded images (or picture). H.264/AVC and HEVC, asmany other video compression standards, divide a picture into a mesh ofrectangles, for each of which a similar block in one of the referencepictures is indicated for inter prediction. The location of theprediction block is coded as a motion vector that indicates the positionof the prediction block relative to the block being coded.

H.264/AVC and HEVC include a concept of picture order count (POC). Avalue of POC is derived for each picture and is non-decreasing withincreasing picture position in output order. POC therefore indicates theoutput order of pictures. POC may be used in the decoding process forexample for implicit scaling of motion vectors in the temporal directmode of bi-predictive slices, for implicitly derived weights in weightedprediction, and for reference picture list initialization. Furthermore,POC may be used in the verification of output order conformance. InH.264/AVC, POC is specified relative to the previous IDR picture or apicture containing a memory management control operation marking allpictures as “unused for reference”.

In H.265/HEVC version 1, picture order count (de)coding and derivation,when no enhancement layers are considered, is carried out as follows:

POC is specified relative to the previous IRAP picture withNoRaslOutputFlag equal to 1. The value of NoRaslOutputFlag is equal to 1for each IDR picture, each BLA picture, and each CRA picture that is thefirst picture in the bitstream in decoding order, is the first picturethat follows an end of sequence NAL unit in decoding order, or hasHandleCraAsBlaFlag equal to 1.

4 to 16 bits of the least significant bits (LSBs) of the POC values areencoded into a bitstream and/or decoded from a bitstream for eachpicture (other than IDR pictures for which the LSB). To be more specificthe LSBs are represented by u(v)-coded slice_pic_order_cnt_lsb syntaxelement, which is present in the slice segment headers (for otherpicture types than IDR pictures).

The number of bits of the slice_pic_order_cnt_lsb syntax element isspecified by the ue(v)-coded log 2_max_pic_order_cnt_lsb_minus4 syntaxelement in the sequence parameter set syntax structure. In log2_max_pic_order_cnt_lsb_minus4 also specifies the value of the variableMaxPicOrderCntLsb that is used in the decoding process for picture ordercount as follows:

MaxPicOrderCntLsb=2^((log 2) ^(_) ^(max) ^(_) ^(pic) ^(_) ^(order) ^(_)^(cnt) ^(_) ^(lsb) ^(_) ^(minus4+4))

The value of log 2_max_pic_order_cnt_lsb_minus4 is in the range of 0 to12, inclusive.

The signaled POC LSB is used to determine whether the POC value of thecurrent picture is smaller or larger than the POC value of the previouspicture in decoding order that has TemporalId equal to 0 and that is nota RASL picture, a RADL picture, or a sub-layer non-reference picture.This previous picture is referred to as prevTid0Pic in the H.265/HEVCdecoding process.

The decoding process of deriving PicOrderCntVal, the picture order countof the current picture, is carried out as follows:

When the current picture is not an TRAP picture with NoRaslOutputFlagequal to 1, the variables prevPicOrderCntLsb and prevPicOrderCntMsb arederived as follows:

-   -   Let prevTid0Pic be the previous picture in decoding order that        has TemporalId equal to 0 and that is not a RASL picture, a RADL        picture, or a sub-layer non-reference picture.    -   The variable prevPicOrderCntLsb is set equal to        slice_pic_order_cnt_lsb of prevTid0Pic.    -   The variable prevPicOrderCntMsb is set equal to PicOrderCntMsb        of prevTid0Pic.

The variable PicOrderCntMsb of the current picture is derived asfollows:

-   -   If the current picture is an IRAP picture with NoRaslOutputFlag        equal to 1, PicOrderCntMsb is set equal to 0.    -   Otherwise, PicOrderCntMsb is derived as follows:

- if( ( slice_pic_order_cnt_lsb < prevPicOrderCntLsb ) && ( (prevPicOrderCntLsb − slice_pic_order_cnt_lsb ) >= ( MaxPicOrderCntLsb /2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb + MaxPicOrderCntLsb else if((slice_pic_order_cnt_lsb > prevPicOrderCntLsb ) && ( (slice_pic_order_cnt_lsb − prevPicOrderCntLsb ) > ( MaxPicOrderCntLsb / 2) ) ) PicOrderCntMsb = prevPicOrderCntMsb − MaxPicOrderCntLsb elsePicOrderCntMsb = prevPicOrderCntMsb

PicOrderCntVal is derived as follows:PicOrderCntVal=PicOrderCntMsb+slice_pic_order_cnt_lsb

The process above has an impact that all IDR pictures will havePicOrderCntVal equal to 0 since slice_pic_order_cnt_lsb is inferred tobe 0 for IDR pictures and prevPicOrderCntLsb and prevPicOrderCntMsb areboth set equal to 0.

The function PicOrderCnt(picX) is specified as follows:PicOrderCnt(picX)=PicOrderCntVal of the picture picX

The function DiffPicOrderCnt(picA, picB) is specified as follows:DiffPicOrderCnt(picA,picB)=PicOrderCnt(picA)−PicOrderCnt(picB)

In H.265/HEVC, it is required that the value of PicOrderCntVal is in therange of −2³¹ to 2³¹−1, inclusive. In one CVS, the PicOrderCntVal valuesfor any two coded pictures are required not be the same. Furthermore, inH.265/HEVC it is required that the bitstream does not contain data thatresult in values of DiffPicOrderCnt(picA, picB) used in the decodingprocess that are not in the range of −2¹⁵ to 2¹⁵−1, inclusive.

Inter prediction process may be characterized using one or more of thefollowing factors.

The Accuracy of Motion Vector Representation.

For example, motion vectors may be of quarter-pixel accuracy, and samplevalues in fractional-pixel positions may be obtained using a finiteimpulse response (FIR) filter.

Block Partitioning for Inter Prediction.

Many coding standards, including H.264/AVC and HEVC, allow selection ofthe size and shape of the block for which a motion vector is applied formotion-compensated prediction in the encoder, and indicating theselected size and shape in the bitstream so that decoders can reproducethe motion-compensated prediction done in the encoder.

Number of Reference Pictures for Inter Prediction.

The sources of inter prediction are previously decoded pictures. Manycoding standards, including H.264/AVC and HEVC, enable storage ofmultiple reference pictures for inter prediction and selection of theused reference picture on a block basis. For example, reference picturesmay be selected on macroblock or macroblock partition basis in H.264/AVCand on PU or CU basis in HEVC. Many coding standards, such as H.264/AVCand HEVC, include syntax structures in the bitstream that enabledecoders to create one or more reference picture lists. A referencepicture index to a reference picture list may be used to indicate whichone of the multiple reference pictures is used for inter prediction fora particular block. A reference picture index may be coded by an encoderinto the bitstream in some inter coding modes or it may be derived (byan encoder and a decoder) for example using neighboring blocks in someother inter coding modes.

Motion Vector Prediction.

In order to represent motion vectors efficiently in bitstreams, motionvectors may be coded differentially with respect to a block-specificpredicted motion vector. In many video codecs, the predicted motionvectors are created in a predefined way, for example by calculating themedian of the encoded or decoded motion vectors of the adjacent blocks.Another way to create motion vector predictions, sometimes referred toas advanced motion vector prediction (AMVP), is to generate a list ofcandidate predictions from adjacent blocks and/or co-located blocks intemporal reference pictures and signalling the chosen candidate as themotion vector predictor. In addition to predicting the motion vectorvalues, the reference index of previously coded/decoded picture can bepredicted. The reference index may be predicted e.g. from adjacentblocks and/or co-located blocks in temporal reference picture.Differential coding of motion vectors may be disabled across sliceboundaries.

Multi-Hypothesis Motion-Compensated Prediction.

H.264/AVC and HEVC enable the use of a single prediction block in Pslices (herein referred to as uni-predictive slices) or a linearcombination of two motion-compensated prediction blocks forbi-predictive slices, which are also referred to as B slices. Individualblocks in B slices may be bi-predicted, uni-predicted, orintra-predicted, and individual blocks in P slices may be uni-predictedor intra-predicted. The reference pictures for a bi-predictive picturemay not be limited to be the subsequent picture and the previous picturein output order, but rather any reference pictures may be used. In manycoding standards, such as H.264/AVC and HEVC, one reference picturelist, referred to as reference picture list 0, is constructed for Pslices, and two reference picture lists, list 0 and list 1, areconstructed for B slices. For B slices, when prediction in forwarddirection may refer to prediction from a reference picture in referencepicture list 0, and prediction in backward direction may refer toprediction from a reference picture in reference picture list 1, eventhough the reference pictures for prediction may have any decoding oroutput order relation to each other or to the current picture.

Weighted Prediction.

Many coding standards use a prediction weight of 1 for prediction blocksof inter (P) pictures and 0.5 for each prediction block of a B picture(resulting into averaging). H.264/AVC allows weighted prediction forboth P and B slices. In implicit weighted prediction, the weights areproportional to picture order counts (POC), while in explicit weightedprediction, prediction weights are explicitly indicated.

In many video codecs, the prediction residual after motion compensationis first transformed with a transform kernel (like DCT) and then coded.The reason for this is that often there still exists some correlationamong the residual and transform can in many cases help reduce thiscorrelation and provide more efficient coding.

In HEVC, each PU has prediction information associated with it definingwhat kind of a prediction is to be applied for the pixels within that PU(e.g. motion vector information for inter predicted PUs and intraprediction directionality information for intra predicted PUs).Similarly each TU is associated with information describing theprediction error decoding process for the samples within the TU(including e.g. DCT coefficient information). It may be signaled at CUlevel whether prediction error coding is applied or not for each CU. Inthe case there is no prediction error residual associated with the CU,it can be considered there are no TUs for the CU.

In some coding formats and codecs, a distinction is made betweenso-called short-term and long-term reference pictures. This distinctionmay affect some decoding processes such as motion vector scaling in thetemporal direct mode or implicit weighted prediction. If both of thereference pictures used for the temporal direct mode are short-termreference pictures, the motion vector used in the prediction may bescaled according to the picture order count difference between thecurrent picture and each of the reference pictures. However, if at leastone reference picture for the temporal direct mode is a long-termreference picture, default scaling of the motion vector may be used, forexample scaling the motion to half may be used. Similarly, if ashort-term reference picture is used for implicit weighted prediction,the prediction weight may be scaled according to the POC differencebetween the POC of the current picture and the POC of the referencepicture. However, if a long-term reference picture is used for implicitweighted prediction, a default prediction weight may be used, such as0.5 in implicit weighted prediction for bi-predicted blocks.

Some video coding formats, such as H.264/AVC, include the frame_numsyntax element, which is used for various decoding processes related tomultiple reference pictures. In H.264/AVC, the value of frame_num forIDR pictures is 0. The value of frame_num for non-IDR pictures is equalto the frame_num of the previous reference picture in decoding orderincremented by 1 (in modulo arithmetic, i.e., the value of frame_numwrap over to 0 after a maximum value of frame_num).

A syntax structure for (decoded) reference picture marking may exist ina video coding system. For example, when the decoding of the picture hasbeen completed, the decoded reference picture marking syntax structure,if present, may be used to adaptively mark pictures as “unused forreference” or “used for long-term reference”. If the decoded referencepicture marking syntax structure is not present and the number ofpictures marked as “used for reference” can no longer increase, asliding window reference picture marking may be used, which basicallymarks the earliest (in decoding order) decoded reference picture asunused for reference.

H.264/AVC specifies the process for decoded reference picture marking inorder to control the memory consumption in the decoder. The maximumnumber of reference pictures used for inter prediction, referred to asM, is determined in the sequence parameter set. When a reference pictureis decoded, it is marked as “used for reference”. If the decoding of thereference picture caused more than M pictures marked as “used forreference”, at least one picture is marked as “unused for reference”.There are two types of operation for decoded reference picture marking:adaptive memory control and sliding window. The operation mode fordecoded reference picture marking is selected on picture basis. Theadaptive memory control enables explicit signaling which pictures aremarked as “unused for reference” and may also assign long-term indicesto short-term reference pictures. The adaptive memory control mayrequire the presence of memory management control operation (MMCO)parameters in the bitstream. MMCO parameters may be included in adecoded reference picture marking syntax structure. If the slidingwindow operation mode is in use and there are M pictures marked as “usedfor reference”, the short-term reference picture that was the firstdecoded picture among those short-term reference pictures that aremarked as “used for reference” is marked as “unused for reference”. Inother words, the sliding window operation mode results intofirst-in-first-out buffering operation among short-term referencepictures.

One of the memory management control operations in H.264/AVC causes allreference pictures except for the current picture to be marked as“unused for reference”. An instantaneous decoding refresh (IDR) picturecontains only intra-coded slices and causes a similar “reset” ofreference pictures.

In HEVC, reference picture marking syntax structures and relateddecoding processes have been replaced with a reference picture set (RPS)syntax structure and decoding process are used instead for a similarpurpose. A reference picture set valid or active for a picture includesall the reference pictures used as reference for the picture and all thereference pictures that are kept marked as “used for reference” for anysubsequent pictures in decoding order. There are six subsets of thereference picture set, which are referred to as RefPicSetStCurr0 (a.k.a.RefPicSetStCurrBefore), RefPicSetStCurr1 (a.k.a. RefPicSetStCurrAfter),RefPicSetStFoll0, RefPicSetStFoll1, RefPicSetLtCurr, andRefPicSetLtFoll. RefPicSetStFoll0 and RefPicSetStFoll1 may also beconsidered to form jointly one subset RefPicSetStFoll. The notation ofthe six subsets is as follows. “Curr” refers to the reference picturesthat are included in the reference picture lists of the current pictureand hence may be used as inter prediction reference for the currentpicture. “Foll” refers to reference pictures that are not included inthe reference picture lists of the current picture but may be used insubsequent pictures in decoding order as reference pictures. “St” refersto short-term reference pictures, which may generally be identifiedthrough a certain number of least significant bits of their POC value.“Lt” refers to long-term reference pictures, which are specificallyidentified and generally have a greater difference of POC valuesrelative to the current picture than what can be represented by thementioned certain number of least significant bits. “0” refers to thosereference pictures that have a smaller POC value than that of thecurrent picture. “1” refers to those reference pictures that have agreater POC value than that of the current picture. RefPicSetStCurr0,RefPicSetStCurr1, RefPicSetStFoll0 and RefPicSetStFoll1 are collectivelyreferred to as the short-term subset of the reference picture set.RefPicSetLtCurr and RefPicSetLtFoll are collectively referred to as thelong-term subset of the reference picture set.

In HEVC, a reference picture set may be specified in a picture parameterset and taken into use in the slice header through an index to thereference picture set. A reference picture set may also be specified ina slice header. A long-term subset of a reference picture set isgenerally specified only in a slice header, while the short-term subsetsof the same reference picture set may be specified in the pictureparameter set or slice header. A reference picture set may be codedindependently or may be predicted from another reference picture set(known as inter-RPS prediction). When a reference picture set isindependently coded, the syntax structure includes up to three loopsiterating over different types of reference pictures; short-termreference pictures with lower POC value than the current picture,short-term reference pictures with higher POC value than the currentpicture, and long-term reference pictures. Each loop entry specifies apicture to be marked as “used for reference”. In general, the picture isspecified with a differential POC value. The inter-RPS predictionexploits the fact that the reference picture set of the current picturecan be predicted from the reference picture set of a previously decodedpicture. This is because all the reference pictures of the currentpicture are either reference pictures of the previous picture or thepreviously decoded picture itself. It is only necessary to indicatewhich of these pictures should be reference pictures and be used for theprediction of the current picture. In both types of reference pictureset coding, a flag (used_by_curr_pic_X_flag) is additionally sent foreach reference picture indicating whether the reference picture is usedfor reference by the current picture (included in a *Curr list) or not(included in a *Foll list). Pictures that are included in the referencepicture set used by the current slice are marked as “used forreference”, and pictures that are not in the reference picture set usedby the current slice are marked as “unused for reference”. If thecurrent picture is an IDR picture, RefPicSetStCurr0, RefPicSetStCurr1,RefPicSetStFoll0, RefPicSetStFoll1, RefPicSetLtCurr, and RefPicSetLtFollare all set to empty.

A Decoded Picture Buffer (DPB) may be used in the encoder and/or in thedecoder. There are two reasons to buffer decoded pictures, forreferences in inter prediction and for reordering decoded pictures intooutput order. As H.264/AVC and HEVC provide a great deal of flexibilityfor both reference picture marking and output reordering, separatebuffers for reference picture buffering and output picture buffering maywaste memory resources. Hence, the DPB may include a unified decodedpicture buffering process for reference pictures and output reordering.A decoded picture may be removed from the DPB when it is no longer usedas a reference and is not needed for output.

In many coding modes of H.264/AVC and HEVC, the reference picture forinter prediction is indicated with an index to a reference picture list.The index may be coded with CABAC or variable length coding. In general,the smaller the index is, the shorter the corresponding syntax elementmay become. In H.264/AVC and HEVC, two reference picture lists(reference picture list 0 and reference picture list 1) are generatedfor each bi-predictive (B) slice, and one reference picture list(reference picture list 0) is formed for each inter-coded (P) slice. Inaddition, for a B slice in a draft version of the HEVC standard, acombined list (List C) may be constructed after the final referencepicture lists (List 0 and List 1) have been constructed. The combinedlist may be used for uni-prediction (also known as uni-directionalprediction) within B slices. However, in the final H.265/HEVC standard,no combined list is constructed.

A reference picture list, such as the reference picture list 0 and thereference picture list 1, may be constructed in two steps: First, aninitial reference picture list is generated. The initial referencepicture list may be generated for example on the basis of frame_num,POC, temporal_id, or information on the prediction hierarchy such as aGOP structure, or any combination thereof. Second, the initial referencepicture list may be reordered by reference picture list reordering(RPLR) commands, also known as reference picture list modificationsyntax structure, which may be contained in slice headers. The RPLRcommands indicate the pictures that are ordered to the beginning of therespective reference picture list. This second step may also be referredto as the reference picture list modification process, and the RPLRcommands may be included in a reference picture list modification syntaxstructure. If reference picture sets are used, the reference picturelist 0 may be initialized to contain RefPicSetStCurr0 first, followed byRefPicSetStCurr1, followed by RefPicSetLtCurr. Reference picture list 1may be initialized to contain RefPicSetStCurr1 first, followed byRefPicSetStCurr0. The initial reference picture lists may be modifiedthrough the reference picture list modification syntax structure, wherepictures in the initial reference picture lists may be identifiedthrough an entry index to the list.

Since multiview video provides encoders and decoders the possibility toutilize inter-view redundancy, decoded inter-view frames may be includedin the reference picture list(s) as well.

Examples of motion vector prediction schemes and related coding modesare provided in the next paragraphs.

In addition to the motion-compensated macroblock modes for which adifferential motion vector is coded, a P macroblock may also be coded inthe so-called P_Skip type in H.264/AVC. For this coding type, nodifferential motion vector, reference index, or quantized predictionerror signal is coded into the bitstream. The reference picture of amacroblock coded with the P_Skip type has index 0 in reference picturelist 0. The motion vector used for reconstructing the P_Skip macroblockis obtained using median motion vector prediction for the macroblockwithout any differential motion vector being added. P_Skip may bebeneficial for compression efficiency particularly in areas where themotion field is smooth.

In B slices of H.264/AVC, four different types of inter prediction aresupported: uni-predictive from reference picture list 0, uni-directionalfrom reference picture list 1, bi-predictive, direct prediction, andB_skip. The type of inter prediction can be selected separately for eachmacroblock partition. B slices utilize a similar macroblock partitioningas P slices. For a bi-predictive macroblock partition, the predictionsignal is formed by a weighted average of motion-compensated list 0 andlist 1 prediction signals. Reference indices, motion vector differences,as well as quantized prediction error signal may be coded foruni-predictive and bi-predictive B macroblock partitions.

Two direct modes are included in H.264/AVC, temporal direct and spatialdirect, and one of them can be selected into use for a slice in a sliceheader, although their use may be constrained further for example inprofiles or alike. In the temporal direct mode, the reference index forreference picture list 1 is set to 0 and the reference index forreference picture list 0 is set to point to the reference picture thatis used in the co-located block (compared to the current block cb) ofthe reference picture having index 0 in the reference picture list 1 ifthat reference picture is available, or set to 0 if that referencepicture is not available. The motion vector predictor for cb isessentially derived by considering the motion information within aco-located block of the reference picture having index 0 in referencepicture list 1. Motion vector predictors for a temporal direct block arederived by scaling a motion vector from the co-located block. Thescaling is proportional to picture order count differences between thecurrent picture and the reference pictures associated with the inferredreference indexes in list 0 and list 1, and by selecting the sign forthe motion vector predictor depending on which reference picture list itis using.

In spatial direct mode of H.264/AVC, motion information of spatiallyadjacent blocks is exploited. Motion vector prediction in spatial directmode can be divided into three steps: reference index determination,determination of uni- or bi-prediction, and motion vector prediction. Inthe first step, the reference picture with the minimum non-negativereference index (i.e., non-intra block) is selected from each ofreference picture list 0 and reference picture list 1 of the neighboringblocks A, B, and C (where A is the adjacent block on the left of thecurrent block, B is the adjacent block above the current block and C isthe adjacent block on the top-right side of the current block). If nonon-negative reference index exists in reference picture list 0 of theneighboring blocks A, B, and C, and likewise no non-negative referenceindex exists in reference picture list 1 of the neighboring blocks A, B,and C, reference index 0 is selected for both reference picture lists.

The use of uni- or bi-prediction for H.264/AVC spatial direct mode isdetermined as follows: If a minimum non-negative reference index forboth reference picture lists was found in the reference indexdetermination step, bi-prediction is used. If a minimum non-negativereference index for either but not both of reference picture list 0 orreference picture list 1 was found in the reference index determinationstep, uni-prediction from either reference picture list 0 or referencepicture list 1, respectively, is used.

In the motion vector prediction for H.264/AVC spatial direct mode,certain conditions, such as whether a negative reference index wasconcluded in the first step, are checked and, if fulfilled, a zeromotion vector is determined. Otherwise, the motion vector predictor isderived similarly to the motion vector predictor of P blocks using themotion vectors of spatially adjacent blocks A, B, and C.

No motion vector differences or reference indices are present in thebitstream for a direct mode block in H.264/AVC, while quantizedprediction error signal may be coded and present therefore present inthe bitstream. A B_skip macroblock mode in H.264/AVC is similar to thedirect mode but no prediction error signal is coded and included in thebitstream.

H.265/HEVC includes two motion vector prediction schemes, namely theadvanced motion vector prediction (AMVP) and the merge mode. In the AMVPor the merge mode, a list of motion vector candidates is derived for aPU. There are two kinds of candidates: spatial candidates and temporalcandidates, where temporal candidates may also be referred to as TMVPcandidates.

A candidate list derivation may be performed for example as follows,while it should be understood that other possibilities exist forcandidate list derivation. If the occupancy of the candidate list is notat maximum, the spatial candidates are included in the candidate listfirst if they are available and not already exist in the candidate list.After that, if occupancy of the candidate list is not yet at maximum, atemporal candidate is included in the candidate list. If the number ofcandidates still does not reach the maximum allowed number, the combinedbi-predictive candidates (for B slices) and a zero motion vector areadded in. After the candidate list has been constructed, the encoderdecides the final motion information from candidates for example basedon a rate-distortion optimization (RDO) decision and encodes the indexof the selected candidate into the bitstream. Likewise, the decoderdecodes the index of the selected candidate from the bitstream,constructs the candidate list, and uses the decoded index to select amotion vector predictor from the candidate list.

In H.265/HEVC, AMVP and the merge mode may be characterized as follows.In AMVP, the encoder indicates whether uni-prediction or bi-predictionis used and which reference pictures are used as well as encodes amotion vector difference. In the merge mode, only the chosen candidatefrom the candidate list is encoded into the bitstream indicating thecurrent prediction unit has the same motion information as that of theindicated predictor. Thus, the merge mode creates regions composed ofneighboring prediction blocks sharing identical motion information,which is only signaled once for each region. Another difference betweenAMVP and the merge mode in H.265/HEVC is that the maximum number ofcandidates of AMVP is 2 while that of the merge mode is 5.

The advanced motion vector prediction may operate for example asfollows, while other similar realizations of advanced motion vectorprediction are also possible for example with different candidateposition sets and candidate locations with candidate position sets. Twospatial motion vector predictors (MVPs) may be derived and a temporalmotion vector predictor (TMVP) may be derived. They may be selectedamong the positions: three spatial motion vector predictor candidatepositions located above the current prediction block (B0, B1, B2) andtwo on the left (A0, A1). The first motion vector predictor that isavailable (e.g. resides in the same slice, is inter-coded, etc.) in apre-defined order of each candidate position set, (B0, B1, B2) or (A0,A1), may be selected to represent that prediction direction (up or left)in the motion vector competition. A reference index for the temporalmotion vector predictor may be indicated by the encoder in the sliceheader (e.g. as a collocated_ref_idx syntax element). The motion vectorobtained from the co-located picture may be scaled according to theproportions of the picture order count differences of the referencepicture of the temporal motion vector predictor, the co-located picture,and the current picture. Moreover, a redundancy check may be performedamong the candidates to remove identical candidates, which can lead tothe inclusion of a zero motion vector in the candidate list. The motionvector predictor may be indicated in the bitstream for example byindicating the direction of the spatial motion vector predictor (up orleft) or the selection of the temporal motion vector predictorcandidate.

The merging/merge mode/process/mechanism may operate for example asfollows, while other similar realizations of the merge mode are alsopossible for example with different candidate position sets andcandidate locations with candidate position sets.

In the merging/merge mode/process/mechanism, where all the motioninformation of a block/PU is predicted and used without anymodification/correction. The aforementioned motion information for a PUmay comprise one or more of the following: 1) The information whether‘the PU is uni-predicted using only reference picture list0’ or ‘the PUis uni-predicted using only reference picture list 1’ or ‘the PU isbi-predicted using both reference picture list0 and list 1’; 2) Motionvector value corresponding to the reference picture list0, which maycomprise a horizontal and vertical motion vector component; 3) Referencepicture index in the reference picture list0 and/or an identifier of areference picture pointed to by the Motion vector corresponding toreference picture list 0, where the identifier of a reference picturemay be for example a picture order count value, a layer identifier value(for inter-layer prediction), or a pair of a picture order count valueand a layer identifier value; 4) Information of the reference picturemarking of the reference picture, e.g. information whether the referencepicture was marked as “used for short-term reference” or “used forlong-term reference”; 5)-7) The same as 2)-4), respectively, but forreference picture list 1.

Similarly, predicting the motion information is carried out using themotion information of adjacent blocks and/or co-located blocks intemporal reference pictures. A list, often called as a merge list, maybe constructed by including motion prediction candidates associated withavailable adjacent/co-located blocks and the index of selected motionprediction candidate in the list is signalled and the motion informationof the selected candidate is copied to the motion information of thecurrent PU. When the merge mechanism is employed for a whole CU and theprediction signal for the CU is used as the reconstruction signal, i.e.prediction residual is not processed, this type of coding/decoding theCU is typically named as skip mode or merge based skip mode. In additionto the skip mode, the merge mechanism may also be employed forindividual PUs (not necessarily the whole CU as in skip mode) and inthis case, prediction residual may be utilized to improve predictionquality. This type of prediction mode is typically named as aninter-merge mode.

One of the candidates in the merge list and/or the candidate list forAMVP or any similar motion vector candidate list may be a TMVP candidateor alike, which may be derived from the collocated block within anindicated or inferred reference picture, such as the reference pictureindicated for example in the slice header. In HEVC, the referencepicture list to be used for obtaining a collocated partition is chosenaccording to the collocated_from_l0_flag syntax element in the sliceheader. When the flag is equal to 1, it specifies that the picture thatcontains the collocated partition is derived from list 0, otherwise thepicture is derived from list 1. When collocated_from_l0_flag is notpresent, it is inferred to be equal to 1. The collocated_ref_idx in theslice header specifies the reference index of the picture that containsthe collocated partition. When the current slice is a P slice,collocated_ref_idx refers to a picture in list 0. When the current sliceis a B slice, collocated_ref_idx refers to a picture in list 0 ifcollocated_from_l0 is 1, otherwise it refers to a picture in list 1.collocated_ref_idx always refers to a valid list entry, and theresulting picture is the same for all slices of a coded picture. Whencollocated_ref_idx is not present, it is inferred to be equal to 0.

In HEVC the so-called target reference index for temporal motion vectorprediction in the merge list is set as 0 when the motion coding mode isthe merge mode. When the motion coding mode in HEVC utilizing thetemporal motion vector prediction is the advanced motion vectorprediction mode, the target reference index values are explicitlyindicated (e.g. per each PU).

In HEVC, the availability of a candidate predicted motion vector (PMV)may be determined as follows (both for spatial and temporal candidates)(SRTP=short-term reference picture, LRTP=long-term reference picture):

reference picture for reference picture for candidate PMV targetreference index candidate PMV availability STRP STRP “available” (andscaled) STRP LTRP “unavailable” LTRP STRP “unavailable” LTRP LTRP“available” (but not scaled)

In HEVC, when the target reference index value has been determined, themotion vector value of the temporal motion vector prediction may bederived as follows: The motion vector PMV at the block that iscollocated with the bottom-right neighbor of the current prediction unitis obtained. The picture where the collocated block resides may be e.g.determined according to the signalled reference index in the sliceheader as described above. If the PMV at location CO is not available,the motion vector PMV at location Cl of the collocated picture isobtained. The determined available motion vector PMV at the co-locatedblock is scaled with respect to the ratio of a first picture order countdifference and a second picture order count difference. The firstpicture order count difference is derived between the picture containingthe co-located block and the reference picture of the motion vector ofthe co-located block. The second picture order count difference isderived between the current picture and the target reference picture. Ifone but not both of the target reference picture and the referencepicture of the motion vector of the collocated block is a long-termreference picture (while the other is a short-term reference picture),the TMVP candidate may be considered unavailable. If both of the targetreference picture and the reference picture of the motion vector of thecollocated block are long-term reference pictures, no POC-based motionvector scaling may be applied.

Motion parameter types or motion information may include but are notlimited to one or more of the following types:

-   -   an indication of a prediction type (e.g. intra prediction,        uni-prediction, bi-prediction) and/or a number of reference        pictures;    -   an indication of a prediction direction, such as inter (a.k.a.        temporal) prediction, inter-layer prediction, inter-view        prediction, view synthesis prediction (VSP), and inter-component        prediction (which may be indicated per reference picture and/or        per prediction type and where in some embodiments inter-view and        view-synthesis prediction may be jointly considered as one        prediction direction) and/or an indication of a reference        picture type, such as a short-term reference picture and/or a        long-term reference picture and/or an inter-layer reference        picture (which may be indicated e.g. per reference picture)    -   a reference index to a reference picture list and/or any other        identifier of a reference picture (which may be indicated e.g.        per reference picture and the type of which may depend on the        prediction direction and/or the reference picture type and which        may be accompanied by other relevant pieces of information, such        as the reference picture list or alike to which reference index        applies);    -   a horizontal motion vector component (which may be indicated        e.g. per prediction block or per reference index or alike);    -   a vertical motion vector component (which may be indicated e.g.        per prediction block or per reference index or alike);    -   one or more parameters, such as picture order count difference        and/or a relative camera separation between the picture        containing or associated with the motion parameters and its        reference picture, which may be used for scaling of the        horizontal motion vector component and/or the vertical motion        vector component in one or more motion vector prediction        processes (where said one or more parameters may be indicated        e.g. per each reference picture or each reference index or        alike);    -   coordinates of a block to which the motion parameters and/or        motion information applies, e.g. coordinates of the top-left        sample of the block in luma sample units;    -   extents (e.g. a width and a height) of a block to which the        motion parameters and/or motion information applies.

In general, motion vector prediction mechanisms, such as those motionvector prediction mechanisms presented above as examples, may includeprediction or inheritance of certain pre-defined or indicated motionparameters.

A motion field associated with a picture may be considered to compriseof a set of motion information produced for every coded block of thepicture. A motion field may be accessible by coordinates of a block, forexample. A motion field may be used for example in TMVP or any othermotion prediction mechanism where a source or a reference for predictionother than the current (de)coded picture is used.

Different spatial granularity or units may be applied to representand/or store a motion field. For example, a regular grid of spatialunits may be used. For example, a picture may be divided intorectangular blocks of certain size (with the possible exception ofblocks at the edges of the picture, such as on the right edge and thebottom edge). For example, the size of the spatial unit may be equal tothe smallest size for which a distinct motion can be indicated by theencoder in the bitstream, such as a 4×4 block in luma sample units. Forexample, a so-called compressed motion field may be used, where thespatial unit may be equal to a pre-defined or indicated size, such as a16×16 block in luma sample units, which size may be greater than thesmallest size for indicating distinct motion. For example, an HEVCencoder and/or decoder may be implemented in a manner that a motion datastorage reduction (MDSR) or motion field compression is performed foreach decoded motion field (prior to using the motion field for anyprediction between pictures). In an HEVC implementation, MDSR may reducethe granularity of motion data to 16×16 blocks in luma sample units bykeeping the motion applicable to the top-left sample of the 16×16 blockin the compressed motion field. The encoder may encode indication(s)related to the spatial unit of the compressed motion field as one ormore syntax elements and/or syntax element values for example in asequence-level syntax structure, such as a video parameter set or asequence parameter set. In some (de)coding methods and/or devices, amotion field may be represented and/or stored according to the blockpartitioning of the motion prediction (e.g. according to predictionunits of the HEVC standard). In some (de)coding methods and/or devices,a combination of a regular grid and block partitioning may be applied sothat motion associated with partitions greater than a pre-defined orindicated spatial unit size is represented and/or stored associated withthose partitions, whereas motion associated with partitions smaller thanor unaligned with a pre-defined or indicated spatial unit size or gridis represented and/or stored for the pre-defined or indicated units.

In HEVC, several improvements have been made to enable the codec tobetter utilize parallelism, i.e. parallel processing of encoding and/ordecoding tasks, thus more efficiently utilizing modern multi-coreprocessor architectures. While slices in principle can be used toparallelize the decoder, employing slices for parallelism typicallyresults in relatively poor coding efficiency. The concept of wavefrontprocessing has been introduced to HEVC to improve the utilization ofparallelism.

To enable wavefront processing, the encoder and/or the decoder uses theCABAC state of the second CTU of the previous CTU row as the initialCABAC state of the current CTU row. Hence, the processing of the currentCTU row can be started when the processing of the second CTU of theprevious CTU has been finished. Thanks to this property, CTU rows can beprocessed in a parallel fashion. In general, it may be pre-defined e.g.in a coding standard which CTU is used for transferring the entropy(de)coding state of the previous row of CTUs or it may be determined andindicated in the bitstream by the encoder and/or decoded from thebitstream by the decoder.

The wavefront processing in HEVC may be used in two parallelizationapproaches, Wavefront Parallel Processing (WPP) and Overlapped Wavefront(OWF). WPP allows creating picture partitions that can be processed inparallel without incurring high coding losses.

WPP processes rows of coding tree units (CTU) in parallel whilepreserving all coding dependencies. In WPP, entropy coding, predictivecoding as well as in-loop filtering can be applied in a singleprocessing step, which makes the implementations of WPP ratherstraightforward. OWF, in turn, enables to overlap the execution ofconsecutive pictures. When the processing of a coding tree unit row inthe current picture has been finished and no more rows are available,the processing of the next picture can be started instead of waiting forthe current picture to finish.

When a coded picture has been constrained for wavefront processing orwhen tiles have been used, CTU rows or tiles (respectively) may bebyte-aligned in the bitstream and may be preceded by a start code.Additionally, entry points may be provided in the bitstream (e.g. in theslice header) and/or externally (e.g. in a container file). An entrypoint is a byte pointer or a byte count or a similar straightforwardreference mechanism to the start of a CTU row (for wavefront-enabledcoded pictures) or a tile. In HEVC, entry points may be specified usingentry_point_offset_minus1[i] of the slice header. In the HEVC fileformat (ISO/IEC 14496-15), the sub-sample information box may providethe information of entry points. In some scenarios, the use of dependentslice segments may be useful instead of or in addition to entry points.A dependent slice segment may be formed for example for a CTU row when acoded picture is constrained for wavefront processing and consequentlythe start of the dependent slice segment NAL unit may be used todetermine CTU row boundaries.

Many video coding standards specify buffering models and bufferingparameters for bitstreams. Such buffering models may be calledHypothetical Reference Decoder (HRD) or Video Buffer Verifier (VBV). Astandard compliant bitstream complies with the buffering model with aset of buffering parameters specified in the corresponding standard.Such buffering parameters for a bitstream may be explicitly orimplicitly signaled. ‘Implicitly signaled’ means for example that thedefault buffering parameter values according to the profile and levelapply. The HRD/VBV parameters are used, among other things, to imposeconstraints on the bit rate variations of compliant bitstreams.

Video coding standards use variable-bitrate coding, which is caused forexample by the flexibility of the encoder to select adaptively betweenintra and inter coding techniques for compressing video frames. Tohandle fluctuation in the bitrate variation of the compressed video,buffering may be used at the encoder and decoder side. HypotheticalReference Decoder (HRD) may be regarded as a hypothetical decoder modelthat specifies constraints on the variability within conformingbitstreams, conforming NAL unit streams or conforming byte streams thatan encoding process may produce.

A bitstream is compliant if it can be decoded by the HRD without bufferoverflow or, in some cases, underflow. Buffer overflow happens if morebits are to be placed into the buffer when it is full. Buffer underflowhappens if some bits are not in the buffer when said bits are to befetched from the buffer for decoding/playback.

An HRD may be a part of an encoder or operationally connected to theoutput of the encoder. The buffering occupancy and possibly otherinformation of the HRD may be used to control the encoding process. Forexample, if a coded data buffer in the HRD is about to overflow, theencoding bitrate may be reduced for example by increasing a quantizerstep size.

The operation of the HRD may be controlled by HRD parameters, such asbuffer size(s) and initial delay(s). The HRD parameter values may becreated as part of the HRD process included or operationally connectedto encoding. Alternatively, HRD parameters may be generated separatelyfrom encoding, for example in an HRD verifier that processes the inputbitstream with the specified HRD process and generates such HRDparameter values according to which the bitstream in conforming. Anotheruse for an HRD verifier is to verify that a given bitstream and givenHRD parameters actually result into a conforming HRD operation andoutput.

HRD conformance checking may concern for example the following two typesof bitstreams: The first such type of bitstream, called Type Ibitstream, is a NAL unit stream containing only the VCL NAL units andfiller data NAL units for all access units in the bitstream. The secondtype of bitstream, called a Type II bitstream, may contain, in additionto the VCL NAL units and filler data NAL units for all access units inthe bitstream, additional non-VCL NAL units other than filler data NALunits and/or syntax elements such as leading_zero_8 bits, zero_byte,start_code_prefix_one_3 bytes, and trailing_zero_8 bits that form a bytestream from the NAL unit stream.

Two types of HRD parameters (NAL HRD parameters and VCL HRD parameters)may be used. The HRD parameters may be indicated through video usabilityinformation included in the sequence parameter set syntax structure. TheHRD parameters may, for example, include buffer size and input bitrate.

Buffering and picture timing parameters (e.g. included in sequenceparameter sets and picture parameter sets referred to in the VCL NALunits and in buffering period and picture timing SEI messages) may beconveyed to the HRD, in a timely manner, either in the bitstream (bynon-VCL NAL units), or by out-of-band means externally from thebitstream e.g. using a signalling mechanism, such as media parametersincluded in the media line of a session description formatted e.g.according to the Session Description Protocol (SDP). For the purpose ofcounting bits in the HRD, only the appropriate bits that are actuallypresent in the bitstream may be counted. When the content of a non-VCLNAL unit is conveyed for the application by some means other thanpresence within the bitstream, the representation of the content of thenon-VCL NAL unit may or may not use the same syntax as would be used ifthe non-VCL NAL unit were in the bitstream.

The HRD may contain a coded picture buffer (CPB), an instantaneousdecoding process, a decoded picture buffer (DPB), and output cropping.

The CPB may operate on decoding unit basis. A decoding unit may be anaccess unit or it may be a subset of an access unit, such as an integernumber of NAL units. Encoders may determine that decoding units are forexample tiles or CTU rows (when encoding constraints enabling wavefrontprocessing have been applied). When a decoding unit is a subset ofpicture, a lower latency in the encoding and decoding may be achieved.The selection of the decoding unit may be indicated by an encoder in thebitstream. For example, decoding unit SEI messages may indicate decodingunits as follows: The set of NAL units associated with a decoding unitinformation SEI message consists, in decoding order, of the SEI NAL unitcontaining the decoding unit information SEI message and all subsequentNAL units in the access unit up to but not including any subsequent SEINAL unit containing a decoding unit information SEI message. Eachdecoding unit may be required to include at least one VCL NAL unit. Allnon-VCL NAL units associated with a VCL NAL unit may be included in thedecoding unit containing the VCL NAL unit.

The HRD may operate as follows. Data associated with decoding units thatflow into the CPB according to a specified arrival schedule may bedelivered by the Hypothetical Stream Scheduler (HSS). The arrivalschedule may be determined by the encoder and indicated for examplethrough picture timing SEI messages, and/or the arrival schedule may bederived for example based on a bitrate which may be indicated forexample as part of HRD parameters in video usability information (whichmay be included in the sequence parameter set). The HRD parameters invideo usability information may contain many sets of parameters, eachfor different bitrate or delivery schedule. The data associated witheach decoding unit may be removed and decoded instantaneously by theinstantaneous decoding process at CPB removal times. A CPB removal timemay be determined for example using an initial CPB buffering delay,which may be determined by the encoder and indicated for example througha buffering period SEI message, and differential removal delaysindicated for each picture for example though picture timing SEImessages. The initial arrival time (i.e. the arrival time of the firstbit) of the very first decoding unit may be determined to be 0. Theinitial arrival time of any subsequent decoding unit may be determinedto be equal to the final arrival time of the previous decoding unit.Each decoded picture is placed in the DPB. A decoded picture may beremoved from the DPB at the later of the DPB output time or the timethat it becomes no longer needed for inter-prediction reference. Thus,the operation of the CPB of the HRD may comprise timing of decoding unitinitial arrival (when the first bit of the decoding unit enters theCPB), timing of decoding unit removal and decoding of decoding unit,whereas the operation of the DPB of the HRD may comprise removal ofpictures from the DPB, picture output, and current decoded picturemarking and storage.

The operation of an AU-based coded picture buffering in the HRD can bedescribed in a simplified manner as follows. It is assumed that bitsarrive into the CPB at a constant arrival bitrate (when the so-calledlow-delay mode is not in use). Hence, coded pictures or access units areassociated with initial arrival time, which indicates when the first bitof the coded picture or access unit enters the CPB. Furthermore, in thelow-delay mode the coded pictures or access units are assumed to beremoved instantaneously when the last bit of the coded picture or accessunit is inserted into CPB and the respective decoded picture is insertedthen to the DPB, thus simulating instantaneous decoding. This time isreferred to as the removal time of the coded picture or access unit. Theremoval time of the first coded picture of the coded video sequence istypically controlled, for example by the Buffering Period SupplementalEnhancement Information (SEI) message. This so-called initial codedpicture removal delay ensures that any variations of the coded bitrate,with respect to the constant bitrate used to fill in the CPB, do notcause starvation or overflow of the CPB. It is to be understood that theoperation of the CPB is somewhat more sophisticated than what describedhere, having for example the low-delay operation mode and the capabilityto operate at many different constant bitrates. Moreover, the operationof the CPB may be specified differently in different standards.

When the bitstream starts at an IRAP picture, for example as a result ofaccessing a file or stream randomly and starting the decoding from anIRAP picture or tuning into a broadcast, there can be leading pictures(RADL and/or RASL pictures) that follow the IRAP picture in decodingorder and precede it in output order. It is possible to discard or omitthe decoding of these leading pictures following the RAP picture withoutaffecting the decoding operation, as these leading pictures have noeffect on the decoding process of any other pictures.

The buffering period SEI message of HEVC supports indicating two sets ofinitial buffering delay and initial buffering delay offset parameters,which can be signaled for example at an IRAP picture. One set of valuesspecifies the required initial buffering when the leading picturesassociated with the IRAP picture (with which the buffering period SEImessage is associated) are present in the bitstream. The other set ofvalues specifies the required initial buffering when leading picturesare not present in the bitstream or are discarded prior to schedulingthem with HSS and/or inputting them into the CPB. The HRD operation maybe required to be verified with the HRD for both sets of bufferingparameters provided in the buffering period SEI message.

The DPB is used, among other things, to control the required memoryresources for decoding of conforming bitstreams. There are two reasonsto buffer decoded pictures, for references in prediction and forreordering decoded pictures into output order. As H.264/AVC and HEVCprovide a great deal of flexibility for both reference picture markingand output reordering, separate buffers for reference picture bufferingand output picture buffering could have been a waste of memoryresources. Hence, the DPB includes a unified decoded picture bufferingprocess for reference pictures and output reordering. A decoded picturemay be removed from the DPB when it is no longer used as reference andneeded for output.

In output cropping, lines and/or columns of samples may be removed fromdecoded pictures according to a cropping rectangle to form outputpictures. In HEVC, a cropped decoded picture is defined as the result ofcropping a decoded picture based on the conformance cropping windowspecified in the SPS that is referred to by the corresponding codedpicture. Conforming decoders are require to produce numericallyidentical cropped decoded pictures as the decoding process specified inHEVC. The output cropping of HEVC produces cropped decoded pictures.

The HRD may be used to check conformance of bitstreams and decoders.

Bitstream conformance requirements of the HRD may comprise for examplethe following and/or alike. The CPB is required not to overflow(relative to the size which may be indicated for example within HRDparameters of video usability information) or underflow (i.e. theremoval time of a decoding unit cannot be smaller than the arrival timeof the last bit of that decoding unit). The number of pictures in theDPB may be required to be smaller than or equal to a certain maximumnumber, which may be indicated for example in the sequence parameterset. All pictures used as prediction references may be required to bepresent in the DPB. It may be required that the interval for outputtingconsecutive pictures from the DPB is not smaller than a certain minimum.

Decoder conformance requirements of the HRD may comprise for example thefollowing and/or alike A decoder claiming conformance to a specificprofile and level may be required to decode successfully all conformingbitstreams specified for decoder conformance provided that all sequenceparameter sets and picture parameter sets referred to in the VCL NALunits, and appropriate buffering period and picture timing SEI messagesare conveyed to the decoder, in a timely manner, either in the bitstream(by non-VCL NAL units), or by external means. There may be two types ofconformance that can be claimed by a decoder: output timing conformanceand output order conformance.

To check conformance of a decoder, test bitstreams conforming to theclaimed profile and level may be delivered by a hypothetical streamscheduler (HSS) both to the HRD and to the decoder under test (DUT). Allpictures output by the HRD may also be required to be output by the DUTand, for each picture output by the HRD, the values of all samples thatare output by the DUT for the corresponding picture may also be requiredto be equal to the values of the samples output by the HRD.

For output timing decoder conformance, the HSS may operate e.g. withdelivery schedules selected from those indicated in the HRD parametersof video usability information, or with “interpolated” deliveryschedules. The same delivery schedule may be used for both the HRD andDUT. For output timing decoder conformance, the timing (relative to thedelivery time of the first bit) of picture output may be required to bethe same for both HRD and the DUT up to a fixed delay.

For output order decoder conformance, the HSS may deliver the bitstreamto the DUT “by demand” from the DUT, meaning that the HSS delivers bits(in decoding order) only when the DUT requires more bits to proceed withits processing. The HSS may deliver the bitstream to the HRD by one ofthe schedules specified in the bitstream such that the bit rate and CPBsize are restricted. The order of pictures output may be required to bethe same for both HRD and the DUT.

In scalable video coding, a video signal can be encoded into a baselayer and one or more enhancement layers. An enhancement layer mayenhance the temporal resolution (i.e., the frame rate), the spatialresolution, or simply the quality of the video content represented byanother layer or part thereof. Each layer together with all itsdependent layers is one representation of the video signal at a certainspatial resolution, temporal resolution and quality level. In thisdocument, we refer to a scalable layer together with all of itsdependent layers as a “scalable layer representation”. The portion of ascalable bitstream corresponding to a scalable layer representation canbe extracted and decoded to produce a representation of the originalsignal at certain fidelity.

In the following, the term layer is used in context of any type ofscalability, including view scalability and depth enhancements. Anenhancement layer refers to any type of an enhancement, such as SNR,spatial, multiview, depth, bit-depth, chroma format, and/or color gamutenhancement. A base layer also refers to any type of a base operationpoint, such as a base view, a base layer for SNR/spatial scalability, ora texture base view for depth-enhanced video coding.

Scalable video (de)coding may be realized with a concept known assingle-loop decoding, where decoded reference pictures are reconstructedonly for the highest layer being decoded while pictures at lower layersmay not be fully decoded or may be discarded after using them forinter-layer prediction. In single-loop decoding, the decoder performsmotion compensation and full picture reconstruction only for thescalable layer desired for playback (called the “desired layer” or the“target layer”), thereby reducing decoding complexity when compared tomulti-loop decoding. All of the layers other than the desired layer donot need to be fully decoded because all or part of the coded picturedata is not needed for reconstruction of the desired layer. However,lower layers (than the target layer) may be used for inter-layer syntaxor parameter prediction, such as inter-layer motion prediction.Additionally or alternatively, lower layers may be used for inter-layerintra prediction and hence intra-coded blocks of lower layers may haveto be decoded. Additionally or alternatively, inter-layer residualprediction may be applied, where the residual information of the lowerlayers may be used for decoding of the target layer and the residualinformation may need to be decoded or reconstructed. In some codingarrangements, a single decoding loop is needed for decoding of mostpictures, while a second decoding loop may be selectively applied toreconstruct so-called base representations (i.e. decoded base layerpictures), which may be needed as prediction references but not foroutput or display.

Some aspects of the SVC extension of the H.264/AVC standard aredescribed next as an example of a scalable video coding standard.

SVC includes support for coarse-grain quality and spatial scalability(CGS), medium-grain quality scalability (MGS) and temporal scalability.In some scalable video coding schemes, data in an enhancement layer canbe truncated after a certain location, or even at arbitrary positions,where each truncation position may include additional data representingincreasingly enhanced visual quality. Such scalability is referred to asfine-grained (granularity) scalability (FGS). FGS was included in somedraft versions of the SVC standard, but it was eventually excluded fromthe final SVC standard. FGS is subsequently discussed in the context ofsome draft versions of the SVC standard. The scalability provided bythose enhancement layers that cannot be truncated is referred to ascoarse-grained (granularity) scalability (CGS). It collectively includesthe traditional quality (SNR) scalability and spatial scalability. TheSVC standard supports the so-called medium-grained scalability (MGS),where quality enhancement pictures are coded similarly to SNR scalablelayer pictures but indicated by high-level syntax elements similarly toFGS layer pictures, by having the quality_id syntax element greater than0.

SVC uses an inter-layer prediction mechanism, wherein certaininformation can be predicted from layers other than the currentlyreconstructed layer or the next lower layer. Information that could beinter-layer predicted includes intra texture, motion and residual data.Inter-layer motion prediction includes the prediction of block codingmode, header information, etc., wherein motion from the lower layer maybe used for prediction of the higher layer. In case of intra coding, aprediction from surrounding macroblocks or from co-located macroblocksof lower layers is possible. These prediction techniques do not employinformation from earlier coded access units and hence, are referred toas intra prediction techniques. Furthermore, residual data from lowerlayers can also be employed for prediction of the current layer.

SVC allows the use of single-loop decoding. It is enabled by using aconstrained intra texture prediction mode, whereby the inter-layer intratexture prediction can be applied to macroblocks (MBs) for which thecorresponding block of the base layer is located inside intra-MBs. Atthe same time, those intra-MBs in the base layer use constrainedintra-prediction (e.g., having the syntax element“constrained_intra_pred_flag” equal to 1). In single-loop decoding, thedecoder performs motion compensation and full picture reconstructiononly for the scalable layer desired for playback (called the “desiredlayer” or the “target layer”), thereby greatly reducing decodingcomplexity. All of the layers other than the desired layer do not needto be fully decoded because all or part of the data of the MBs not usedfor inter-layer prediction (be it inter-layer intra texture prediction,inter-layer motion prediction or inter-layer residual prediction) is notneeded for reconstruction of the desired layer. A single decoding loopis needed for decoding of most pictures, while a second decoding loop isselectively applied to reconstruct the base representations, which areneeded as prediction references but not for output or display, and arereconstructed only for the so called key pictures (for which“store_ref_base_pic_flag” is equal to 1).

The scalability structure in the SVC draft is characterized by threesyntax elements: “temporal_id,” “dependency_id” and “quality_id.” Thesyntax element “temporal_id” is used to indicate the temporalscalability hierarchy or, indirectly, the frame rate. A scalable layerrepresentation comprising pictures of a smaller maximum “temporal_id”value has a smaller frame rate than a scalable layer representationcomprising pictures of a greater maximum “temporal_id”. A given temporallayer typically depends on the lower temporal layers (i.e., the temporallayers with smaller “temporal_id” values) but does not depend on anyhigher temporal layer. The syntax element “dependency_id” is used toindicate the CGS inter-layer coding dependency hierarchy (which, asmentioned earlier, includes both SNR and spatial scalability). At anytemporal level location, a picture of a smaller “dependency_id” valuemay be used for inter-layer prediction for coding of a picture with agreater “dependency_id” value. The syntax element “quality_id” is usedto indicate the quality level hierarchy of a FGS or MGS layer. At anytemporal location, and with an identical “dependency_id” value, apicture with “quality_id” equal to QL uses the picture with “quality_id”equal to QL-1 for inter-layer prediction. A coded slice with“quality_id” larger than 0 may be coded as either a truncatable FGSslice or a non-truncatable MGS slice.

For simplicity, all the data units (e.g., Network Abstraction Layerunits or NAL units in the SVC context) in one access unit havingidentical value of “dependency_id” are referred to as a dependency unitor a dependency representation. Within one dependency unit, all the dataunits having identical value of “quality_id” are referred to as aquality unit or layer representation.

A base representation, also known as a decoded base picture, is adecoded picture resulting from decoding the Video Coding Layer (VCL) NALunits of a dependency unit having “quality_id” equal to 0 and for whichthe “store_ref_base_pic_flag” is set equal to 1. An enhancementrepresentation, also referred to as a decoded picture, results from theregular decoding process in which all the layer representations that arepresent for the highest dependency representation are decoded.

As mentioned earlier, CGS includes both spatial scalability and SNRscalability. Spatial scalability is initially designed to supportrepresentations of video with different resolutions. For each timeinstance, VCL NAL units are coded in the same access unit and these VCLNAL units can correspond to different resolutions. During the decoding,a low resolution VCL NAL unit provides the motion field and residualwhich can be optionally inherited by the final decoding andreconstruction of the high resolution picture. When compared to oldervideo compression standards, SVC's spatial scalability has beengeneralized to enable the base layer to be a cropped and zoomed versionof the enhancement layer.

MGS quality layers are indicated with “quality_id”. In a draft versionof SVC, quality_id was also used for indicating FGS quality layers. Foreach dependency unit (with the same “dependency_id”), there is a layerwith “quality_id” equal to 0 and there can be other layers with“quality_id” greater than 0. These layers with “quality_id” greater than0 are MGS layers and in a draft version of SVC they were either MGSlayers or FGS layers, depending on whether the slices are coded astruncatable slices.

In the basic form of FGS enhancement layers, only inter-layer predictionis used. Therefore, FGS enhancement layers can be truncated freelywithout causing any error propagation in the decoded sequence. However,the basic form of FGS suffers from low compression efficiency. Thisissue arises because only low-quality pictures are used for interprediction references. It has therefore been proposed that FGS-enhancedpictures be used as inter prediction references. However, this may causeencoding-decoding mismatch, also referred to as drift, when some FGSdata are discarded.

One feature of a draft SVC standard is that the FGS NAL units can befreely dropped or truncated, and a feature of the SVC standard is thatMGS NAL units can be freely dropped (but cannot be truncated) withoutaffecting the decodability of the bitstream. As discussed above, whenthose FGS or MGS data have been used for inter prediction referenceduring encoding, dropping or truncation of the data would result in amismatch between the decoded pictures in the decoder side and in theencoder side. This mismatch is also referred to as drift.

To control drift due to the dropping or truncation of FGS or MGS data,SVC applied the following solution: In a certain dependency unit, a baserepresentation (by decoding only the CGS picture with “quality_id” equalto 0 and all the dependent-on lower layer data) is stored in the decodedpicture buffer. When encoding a subsequent dependency unit with the samevalue of “dependency_id,” all of the NAL units, including FGS or MGS NALunits, use the base representation for inter prediction reference.Consequently, all drift due to dropping or truncation of FGS or MGS NALunits in an earlier access unit is stopped at this access unit. Forother dependency units with the same value of “dependency_id,” all ofthe NAL units use the decoded pictures for inter prediction reference,for high coding efficiency.

Each NAL unit includes in the NAL unit header a syntax element“use_ref_base_pic_flag.” When the value of this element is equal to 1,decoding of the NAL unit uses the base representations of the referencepictures during the inter prediction process. The syntax element“store_ref_base_pic_flag” specifies whether (when equal to 1) or not(when equal to 0) to store the base representation of the currentpicture for future pictures to use for inter prediction.

NAL units with “quality_id” greater than 0 do not contain syntaxelements related to reference picture lists construction and weightedprediction, i.e., the syntax elements “num_ref active_lx_minus1” (x=0 or1), the reference picture list reordering syntax table, and the weightedprediction syntax table are not present. Consequently, the MGS or FGSlayers have to inherit these syntax elements from the NAL units with“quality_id” equal to 0 of the same dependency unit when needed.

In SVC, a reference picture list consists of either only baserepresentations (when “use_ref_base_pic_flag” is equal to 1) or onlydecoded pictures not marked as “base representation” (when“use_ref_base_pic_flag” is equal to 0), but never both at the same time.

In an H.264/AVC bit stream, coded pictures in one coded video sequenceuses the same sequence parameter set, and at any time instance duringthe decoding process, only one sequence parameter set is active. In SVC,coded pictures from different scalable layers may use different sequenceparameter sets. If different sequence parameter sets are used, then, atany time instant during the decoding process, there may be more than oneactive sequence picture parameter set. In the SVC specification, the onefor the top layer is denoted as the active sequence picture parameterset, while the rest are referred to as layer active sequence pictureparameter sets. Any given active sequence parameter set remainsunchanged throughout a coded video sequence in the layer in which theactive sequence parameter set is referred to.

As indicated earlier, MVC is an extension of H.264/AVC. Many of thedefinitions, concepts, syntax structures, semantics, and decodingprocesses of H.264/AVC apply also to MVC as such or with certaingeneralizations or constraints. Some definitions, concepts, syntaxstructures, semantics, and decoding processes of MVC are described inthe following.

An access unit in MVC is defined to be a set of NAL units that areconsecutive in decoding order and contain exactly one primary codedpicture consisting of one or more view components. In addition to theprimary coded picture, an access unit may also contain one or moreredundant coded pictures, one auxiliary coded picture, or other NALunits not containing slices or slice data partitions of a coded picture.The decoding of an access unit results in one decoded picture consistingof one or more decoded view components, when decoding errors, bitstreamerrors or other errors which may affect the decoding do not occur. Inother words, an access unit in MVC contains the view components of theviews for one output time instance.

A view component in MVC is referred to as a coded representation of aview in a single access unit.

Inter-view prediction may be used in MVC and refers to prediction of aview component from decoded samples of different view components of thesame access unit. In MVC, inter-view prediction is realized similarly tointer prediction. For example, inter-view reference pictures are placedin the same reference picture list(s) as reference pictures for interprediction, and a reference index as well as a motion vector are codedor inferred similarly for inter-view and inter reference pictures.

An anchor picture is a coded picture in which all slices may referenceonly slices within the same access unit, i.e., inter-view prediction maybe used, but no inter prediction is used, and all following codedpictures in output order do not use inter prediction from any pictureprior to the coded picture in decoding order. Inter-view prediction maybe used for IDR view components that are part of a non-base view. A baseview in MVC is a view that has the minimum value of view order index ina coded video sequence. The base view can be decoded independently ofother views and does not use inter-view prediction. The base view can bedecoded by H.264/AVC decoders supporting only the single-view profiles,such as the Baseline Profile or the High Profile of H.264/AVC.

In the MVC standard, many of the sub-processes of the MVC decodingprocess use the respective sub-processes of the H.264/AVC standard byreplacing term “picture”, “frame”, and “field” in the sub-processspecification of the H.264/AVC standard by “view component”, “frame viewcomponent”, and “field view component”, respectively. Likewise, terms“picture”, “frame”, and “field” are often used in the following to mean“view component”, “frame view component”, and “field view component”,respectively.

In scalable multiview coding, the same bitstream may contain coded viewcomponents of multiple views and at least some coded view components maybe coded using quality and/or spatial scalability.

Many video encoders utilize the Lagrangian cost function to findrate-distortion optimal coding modes, for example the desired macroblockmode and associated motion vectors. This type of cost function uses aweighting factor or 2 (lambda) to tie together the exact or estimatedimage distortion due to lossy coding methods and the exact or estimatedamount of information required to represent the pixel/sample values inan image area. The Lagrangian cost function may be represented by theequation:C=D+λR

where C is the Lagrangian cost to be minimized, D is the imagedistortion (for example, the mean-squared error between the pixel/samplevalues in original image block and in coded image block) with the modeand motion vectors currently considered, λ is a Lagrangian coefficientand R is the number of bits needed to represent the required data toreconstruct the image block in the decoder (including the amount of datato represent the candidate motion vectors).

There are ongoing standardization activities to specify a multiviewextension of HEVC (which may be referred to as MV-HEVC), adepth-enhanced multiview extension of HEVC (which may be referred to as3D-HEVC), and a scalable extension of HEVC (which may be referred to asSHVC). A multi-loop decoding operation has been envisioned to be used inall these specifications.

In scalable video coding schemes utilizing multi-loop (de)coding,decoded reference pictures for each (de)coded layer may be maintained ina decoded picture buffer (DPB). The memory consumption for DPB maytherefore be significantly higher than that for scalable video codingschemes with single-loop (de)coding operation. However, multi-loop(de)coding may have other advantages, such as relatively few additionalparts compared to single-layer coding.

In scalable video coding with multi-loop decoding, enhanced layers maybe predicted from pictures that had been already decoded in the base(reference) layer. Such pictures may be stored in the DPB of base layerand may be marked as used for reference. In certain circumstances, apicture marked as used for reference may be stored in fast memory, inorder to provide fast random access to its samples, and may remainstored after the picture is supposed to be displayed in order to be usedas reference for prediction. This imposes requirements on memoryorganization. In order to relax such memory requirements, a conventionaldesign in multi-loop multilayer video coding schemes (such as MVC)assumes restricted utilization of inter-layer predictions.Inter-layer/inter-view prediction for enhanced view is allowed from adecoded picture of the base view located at the same access unit, inother word representing the scene at the same time entity. In suchdesigns, the number of reference pictures available for predictingenhanced views is increased by 1 for each reference view.

It has been proposed that in scalable video coding with multi-loop(de)coding operation pictures marked as used for reference need notoriginate from the same access units in all layers. For example, asmaller number of reference pictures may be maintained in an enhancementlayer compared to the base layer. In some embodiments a temporalinter-layer prediction, which may also be referred to as a diagonalinter-layer prediction or diagonal prediction, can be used to improvecompression efficiency in such coding scenarios. In general, diagonalprediction may refer to any prediction where the prediction crosses morethan one scalability domain or scalability type. For example, diagonalprediction may refer to prediction that takes place from a differentcomponent type (e.g. from depth to texture) and from a different timeinstant (e.g. from a picture of a previous access unit in (de)codingorder to a picture in the current access unit).

A decoding process may be specified with reference to a layer identifierlist TargetDecLayerIdList, which specifies the list of layer identifiervalues, such as nuh_layer_id values. The layer identifier values may bein TargetDecLayerIdList in increasing order of the NAL units to bedecoded. TargetDecLayerIdList may include the layer identifiers forlayers that are intended to be output by the decoder as well as all thelayers on which the output layers depend in the decoding process.

Work is ongoing to specify scalable and multiview extensions to the HEVCstandard. The multiview extension of HEVC, referred to as MV-HEVC, issimilar to the MVC extension of H.264/AVC. Similarly to MVC, in MV-HEVC,inter-view reference pictures can be included in the reference picturelist(s) of the current picture being coded or decoded. The scalableextension of HEVC, referred to as SHVC, is planned to be specified sothat it uses multi-loop decoding operation (unlike the SVC extension ofH.264/AVC). SHVC uses a reference index based design, where aninter-layer reference picture can be included in a one or more referencepicture lists of the current picture being coded or decoded (asdescribed above). In an earlier draft of SHVC, another design was alsoinvestigated, which may be referred to as IntraBL or TextureRL, where aspecific coding mode, e.g. in CU level, is used for usingdecoded/reconstructed sample values of a reference layer picture forprediction in an enhancement layer picture. The SHVC development hasconcentrated on development of spatial and coarse grain qualityscalability.

Both MV-HEVC and SHVC use reference-index-based scalability and aretherefore similar to each other. The high-level syntax, semantics anddecoding operation of MV-HEVC and SHVC has been aligned to a greatextent. A difference between SHVC and MV-HEVC is that SHVC enablesspatial scalability and hence includes upsampling of sample arrays andmotion field mapping or upsampling.

It is possible to use many of the same syntax structures, semantics, anddecoding processes for MV-HEVC and reference-index-based SHVC.Furthermore, it is possible to use the same syntax structures,semantics, and decoding processes for depth coding too. Hereafter, termscalable multiview extension of HEVC (SMV-HEVC) is used to refer to acoding process, a decoding process, syntax, and semantics where largelythe same (de)coding tools are used regardless of the scalability typeand where the reference index based approach without changes in thesyntax, semantics, or decoding process below the slice header is used.SMV-HEVC might not be limited to multiview, spatial, and coarse grainquality scalability but may also support other types of scalability,such as depth-enhanced video.

For the enhancement layer coding, the same concepts and coding tools ofHEVC may be used in SHVC, MV-HEVC, and/or SMV-HEVC. However, theadditional inter-layer prediction tools, which employ already coded data(including reconstructed picture samples and motion parameters a.k.amotion information) in reference layer for efficiently coding anenhancement layer, may be integrated to SHVC, MV-HEVC, and/or SMV-HEVCcodec.

An access unit in SHVC, MV-HEVC and SMV-HEVC may be defined as a set ofNAL units that are associated with each other according to a specifiedclassification rule, are consecutive in decoding order, and contain theVCL NAL units of all coded pictures associated with the same output timeand their associated non-VCL NAL units.

In MV-HEVC, SMV-HEVC, and reference index based SHVC solution, the blocklevel syntax and decoding process are not changed for supportinginter-layer texture prediction. Only the high-level syntax has beenmodified (compared to that of HEVC) so that reconstructed pictures(upsampled if necessary) from a reference layer of the same access unitcan be used as the reference pictures for coding the current enhancementlayer picture. The inter-layer reference pictures as well as thetemporal reference pictures are included in the reference picture lists.The signalled reference picture index is used to indicate whether thecurrent Prediction Unit (PU) is predicted from a temporal referencepicture or an inter-layer reference picture. The use of this feature maybe controlled by the encoder and indicated in the bitstream for examplein a video parameter set, a sequence parameter set, a picture parameter,and/or a slice header. The indication(s) may be specific to anenhancement layer, a reference layer, a pair of an enhancement layer anda reference layer, specific TemporalId values, specific picture types(e.g. IRAP pictures), specific slice types (e.g. P and B slices but notI slices), pictures of a specific POC value, and/or specific accessunits, for example. The scope and/or persistence of the indication(s)may be indicated along with the indication(s) themselves and/or may beinferred.

The reference list(s) in MV-HEVC, SMV-HEVC, and a reference index basedSHVC solution may be initialized using a specific process in which theinter-layer reference picture(s), if any, may be included in the initialreference picture list(s). are constructed as follows. For example, thetemporal references may be firstly added into the reference lists (L0,L1) in the same manner as the reference list construction in HEVC. Afterthat, the inter-layer references may be added after the temporalreferences. The inter-layer reference pictures may be for exampleconcluded from the layer dependency information, such as theRefLayerId[i] variable derived from the VPS extension as describedabove. The inter-layer reference pictures may be added to the initialreference picture list L0 if the current enhancement-layer slice is aP-Slice, and may be added to both initial reference picture lists L0 andL1 if the current enhancement-layer slice is a B-Slice. The inter-layerreference pictures may be added to the reference picture lists in aspecific order, which can but need not be the same for both referencepicture lists. For example, an opposite order of adding inter-layerreference pictures into the initial reference picture list 1 may be usedcompared to that of the initial reference picture list 0. For example,inter-layer reference pictures may be inserted into the initialreference picture 0 in an ascending order of nuh_layer_id, while anopposite order may be used to initialize the initial reference picturelist 1.

In the coding and/or decoding process, the inter-layer referencepictures may be treated as a long-term reference pictures.

In SMV-HEVC and a reference index based SHVC solution, inter-layermotion parameter prediction may be performed by setting the inter-layerreference picture as the collocated picture for TMVP derivation. Amotion field mapping process between two layers may be performed forexample to avoid block level decoding process modification in TMVPderivation. A motion field mapping could also be performed for multiviewcoding, but a present draft of MV-HEVC (JCT-3V document JCT3V-E1004)does not include such a process. The use of the motion field mappingfeature may be controlled by the encoder and indicated in the bitstreamfor example in a video parameter set, a sequence parameter set, apicture parameter, and/or a slice header. The indication(s) may bespecific to an enhancement layer, a reference layer, a pair of anenhancement layer and a reference layer, specific TemporalId values,specific picture types (e.g. RAP pictures), specific slice types (e.g. Pand B slices but not I slices), pictures of a specific POC value, and/orspecific access units, for example. The scope and/or persistence of theindication(s) may be indicated along with the indication(s) themselvesand/or may be inferred.

In a motion field mapping process for spatial scalability, the motionfield of the upsampled inter-layer reference picture is attained basedon the motion field of the respective reference layer picture. Themotion parameters (which may e.g. include a horizontal and/or verticalmotion vector value and a reference index) and/or a prediction mode foreach block of the upsampled inter-layer reference picture may be derivedfrom the corresponding motion parameters and/or prediction mode of thecollocated block in the reference layer picture. The block size used forthe derivation of the motion parameters and/or prediction mode in theupsampled inter-layer reference picture may be for example 16×16. The16×16 block size is the same as in HEVC TMVP derivation process wherecompressed motion field of reference picture is used.

In H.265/HEVC, the sps_temporal_mvp_enabled_flag indicates whether theTMVP mechanism may be in use (when the flag is equal to 1) or is not inuse (when the flag is equal to 0) in the HEVC base layer/view (withnuh_layer_id equal to 0). When sps_temporal_mvp_enabled_flag is equal to1, the slice_temporal_mvp_enabled_flag is present in the slice headerand indicates if the TMVP mechanism is in use for the current picture.

There may be “black box” implementations of scalable extensions of HEVC,where the base layer decoding/coding is implemented with an existingHEVC v1 implementation without changes. Such an implementation of baselayer decoding/coding would store motion fields only ifsps_temporal_mvp_enabled_flag is equal to 1.

Base layer motion fields may be used for either or both of the followingtwo purposes: temporal motion vector prediction between pictures of thebase layer and inter-layer motion prediction. If the base layer motionfields are used only for inter-layer motion prediction, the memory usedfor base layer motion fields could be de-allocated or used for otherpurposes after decoding of the access unit has been finished (or, moreaccurately, decoding of all layers within the access unit that may usethe abase layer as motion prediction reference has been finished).However, when sps_temporal_mvp_enabled_flag is used to control thestorage of base layer motion fields, it cannot be used to indicate thatbase layer motion fields are used only for inter-layer motion predictionand not for temporal motion vector prediction within the base layer.

In a textureRL based SHVC solution, the inter-layer texture predictionmay be performed at CU level for which a new prediction mode, named astextureRL mode, is introduced. The collocated upsampled base layer blockis used as the prediction for the enhancement layer CU coded intextureRL mode. For an input CU of the enhancement layer encoder, the CUmode may be determined among intra, inter and textureRL modes, forexample. The use of the textureRL feature may be controlled by theencoder and indicated in the bitstream for example in a video parameterset, a sequence parameter set, a picture parameter, and/or a sliceheader. The indication(s) may be specific to an enhancement layer, areference layer, a pair of an enhancement layer and a reference layer,specific TemporalId values, specific picture types (e.g. RAP pictures),specific slice types (e.g. P and B slices but not I slices), pictures ofa specific POC value, and/or specific access units, for example. Thescope and/or persistence of the indication(s) may be indicated alongwith the indication(s) themselves and/or may be inferred. Furthermore,the textureRL may be selected by the encoder at CU level and may beindicated in the bitstream per each CU for example using a CU level flag(texture_rl_flag) which may be entropy-coded e.g. using context adaptivearithmetic coding (e.g. CABAC).

The residue of textureRL predicted CU may be coded as follows. Thetransform process of textureRL predicted CU may be the same as that forthe intra predicted CU, where a discrete sine transform (DST) is appliedto TU of luma component having 4×4 size and a discrete cosine transform(DCT) is applied to the other type of TUs. Transform coefficient codingof a textureRL-predicted CU may be the same to that of inter predictedCU, where no_residue_flag may be used to indicate whether thecoefficients of the whole CU are skipped.

In a textureRL based SHVC solution, in addition to spatially andtemporally neighboring PUs, the motion parameters of the collocatedreference-layer block may also be used to form the merge candidate list.The base layer merge candidate may be derived at a location collocatedto the central position of the current PU and may be inserted in aparticular location of the merge list, such as as the first candidate inmerge list. In the case of spatial scalability, the reference-layermotion vector may be scaled according to the spatial resolution ratiobetween the two layers. The pruning (duplicated candidates check) may beperformed for each spatially neighboring candidate with collocated baselayer candidate. For the collocated base layer merge candidate andspatial merge candidate derivation, a certain maximum number of mergecandidates may be used; for example four merge candidates may beselected among candidates that are located in six different positions.The temporal merge candidate may be derived in the same manner as donefor HEVC merge list. When the number of candidates does not reach tomaximum number of merge candidates (which may be determined by theencoder and may be indicated in the bitstream and may be assigned to thevariable MaxNumMergeCand), the additional candidates, including combinedbi-predictive candidates and zero merge candidates, may be generated andadded at the end of the merge list, similarly or identically to HEVCmerge list construction.

In some coding and/or decoding arrangements, a reference index basedscalability and a block-level scalability approach, such a textureRLbased approach, may be combined. For example, multiview-video-plus-depthcoding and/or decoding may be performed as follows. A textureRL approachmay be used between the components of the same view. For example, adepth view component may be inter-layer predicted using a textureRLapproach from a texture view component of the same view. A referenceindex based approach may be used for inter-view prediction, and in someembodiments inter-view prediction may be applied only between viewcomponents of the same component type.

Work is also ongoing to specify depth-enhanced video coding extensionsto the HEVC standard, which may be referred to as 3D-HEVC, in whichtexture views and depth views may be coded into a single bitstream wheresome of the texture views may be compatible with HEVC. In other words,an HEVC decoder may be able to decode some of the texture views of sucha bitstream and can omit the remaining texture views and depth views.

Other types of scalability and scalable video coding include bit-depthscalability, where base layer pictures are coded at lower bit-depth(e.g. 8 bits) per luma and/or chroma sample than enhancement layerpictures (e.g. 10 or 12 bits), chroma format scalability, whereenhancement layer pictures provide higher fidelity and/or higher spatialresolution in chroma (e.g. coded in 4:4:4 chroma format) than base layerpictures (e.g. 4:2:0 format), and color gamut scalability, where theenhancement layer pictures have a richer/broader color representationrange than that of the base layer pictures—for example the enhancementlayer may have UHDTV (ITU-R BT.2020) color gamut and the base layer mayhave the ITU-R BT.709 color gamut. Additionally or alternatively, depthenhancement layers or views, providing distance and/or disparityinformation, and/or layers with alpha pictures, providing transparencyinformation, and/or other types of auxiliary picture layers may beprovided as scalable layers. Any number of such other types ofscalability may be realized for example with a reference index basedapproach or a block-based approach e.g. as described above.

Another categorization of scalable coding is based on whether the sameor different coding standard or technology is used as the basis for thebase layer and enhancement layers. Terms hybrid codec scalability orstandards scalability may be used to indicate a scenario where onecoding standard or system is used for some layers, while another codingstandard or system is used for some other layers. For example, the baselayer may be AVC-coded, while one or more enhancement layers may becoded with an HEVC extension, such as SHVC or MV-HEVC. It is possiblethat more than one layer is of a first coding standard or system, suchas AVC or its extensions like MVC, and/or more than one layer is asecond coding standard. It is possible that layers represent more thantwo coding standards. For example, the base layer may be coded with AVC,an enhancement layer may be coded with MVC and represent a non-baseview, and either or both of the previous layers may be enhanced by aspatial or quality scalable layer coded with SHVC.

In many video communication or transmission systems, transportmechanisms and multimedia container file formats there are mechanisms totransmit or store the base layer separately from the enhancementlayer(s). It may be considered that layers are stored in or transmittedthrough separate logical channels. Examples are provided in thefollowing:

1. ISO Base Media File Format (ISOBMFF, ISO/IEC International Standard14496-12): Base layer can be stored as a track and each enhancementlayer can be stored in another track. Similarly, in a hybrid codecscalability case, a non-HEVC-coded base layer can be stored as a track(e.g. of sample entry type ‘avc1’), while the enhancement layer(s) canbe stored as another track which is linked to the base-layer track usingso-called track references.2. Real-time Transport Protocol (RTP): either RTP session multiplexingor synchronization source (SSRC) multiplexing can be used to logicallyseparate different layers.3. MPEG-2 transport stream (TS): Each layer can have a different packetidentifier (PID) value.

Many video communication or transmission systems, transport mechanismsand multimedia container file formats provides means to associate codeddata of separate logical channels, such as of different tracks orsessions, with each other. For example, there are mechanisms toassociate coded data of the same access unit together. For example,decoding or output times may be provided in the container file format ortransport mechanism, and coded data with the same decoding or outputtime may be considered to form an access unit.

A way of categorizing different types of prediction is to consideracross which domains or scalability types the prediction crosses. Thiscategorization may lead into one or more of the following types ofprediction, which may also sometimes be referred to as predictiondirections:

-   -   Temporal prediction e.g. of sample values or motion vectors from        an earlier picture usually of the same scalability layer, view        and component type (texture or depth).    -   Inter-view prediction (which may be also referred to as        cross-view prediction) referring to prediction taking place        between view components usually of the same time instant or        access unit and the same component type.    -   Inter-layer prediction referring to prediction taking place        between layers usually of the same time instant, of the same        component type, and of the same view.    -   Inter-component prediction may be defined to comprise prediction        of syntax element values, sample values, variable values used in        the decoding process, or anything alike from a component picture        of one type to a component picture of another type. For example,        inter-component prediction may comprise prediction of a texture        view component from a depth view component, or vice versa.

Prediction approaches using image information from a previously codedimage can also be called as inter prediction methods. Inter predictionmay sometimes be considered to only include motion-compensated temporalprediction, while it may sometimes be considered to include all types ofprediction where a reconstructed/decoded block of samples is used asprediction source, therefore including conventional inter-viewprediction for example. Inter prediction may be considered to compriseonly sample prediction but it may alternatively be considered tocomprise both sample and syntax prediction. As a result of syntax andsample prediction, a predicted block of pixels of samples may beobtained.

If the prediction, such as predicted variable values and/or predictionblocks, is not refined by the encoder using any form of prediction erroror residual coding, prediction may be referred to as inheritance. Forexample, in the merge mode of HEVC, the prediction motion information isnot refined e.g. by (de)coding motion vector differences, and hence themerge mode may be considered as an example of motion informationinheritance.

Video coding schemes may utilize a prediction scheme between pictures.As discussed, prediction may be performed in the encoder for examplethrough a process of block partitioning and block matching between acurrently coded block (Cb) in the current picture and a reference block(Rb) in the picture which is selected as a reference. Thereforeparameters of such prediction can be defined as motion information (MI)comprising for example on or more of the following: spatial coordinatesof the Cb (e.g. coordinates of the top-left pixel of the Cb); areference index refIdx or similar which specifies the picture in thereference picture list which is selected as reference picture; a motionvector (MV) specifying displacement between the spatial coordinated ofthe Cb and Rb in the reference picture; and the size and shape of themotion partition (the size and shape of the matching block).

A motion field associated with a picture may be considered to comprise aset of motion information produced for every coded block of the picture.A motion field may be accessible by coordinates of a block, for example.A motion field may be used for example in Temporal motion vectorprediction or any other motion prediction mechanism where a source or areference for prediction other than the current decoded/coded picture isused.

Video coding schemes may utilize a temporal motion vector predictionscheme, such as the temporal direct mode in H.264/AVC or the temporalmotion vector predictor (TMVP) candidate in the merge and AVMP modes ofH.265/HEVC. In a temporal motion vector prediction scheme, at least asubset of the motion information of another picture is used to derivemotion information or motion information predictor(s) for the currentpicture. Temporal motion vector prediction therefore requires storage ofmotion information of reference pictures.

In H.265/HEVC, the sequence parameter set includes thesps_temporal_mvp_enabled_flag syntax element, which indicates if theslice header includes the slice_temporal_mvp_enabled flag. Ifsps_temporal_mvp_enabled_flag is equal to 0, no temporal motion vectorpredictors are used in the coded video sequence.slice_temporal_mvp_enabled_flag specifies whether temporal motion vectorpredictors can be used for inter prediction. Whenslice_temporal_mvp_enabled_flag is equal to 1, there are syntax elementsin the slice header that identify the collocated picture used to derivethe temporal motion vector predictors.

Temporal motion vector prediction can also be used in scalable videocoding when a motion field of an inter-layer reference picture is usedto predict or derive motion information of the current picture.

Motion field mapping may be used for example when an inter-layerreference picture is of different spatial resolution than the currentpicture. In a motion field mapping process for spatial scalability, themotion field of the upsampled inter-layer reference picture is attainedbased on the motion field of the respective reference layer picture. Themotion parameters (which may e.g. include a horizontal and/or verticalmotion vector value and a reference index) and/or a prediction mode foreach block of the upsampled inter-layer reference picture may be derivedfrom the corresponding motion parameters and/or prediction mode of thecollocated block in the reference layer picture.

The storage of motion information may be performed for example on thebasis of the minimum size of a motion partition, e.g. 4×4 (of lumasamples) in the case of H.264/AVC. In another example, the spatialgranularity of motion information may be pre-defined for example in acoding standard and the coded motion information may be resampled orconverted to that spatial granularity. For example, motion informationcan be stored for 16×16 blocks (of luma samples) in H.265/HEVC.

In scalable and/or multiview video coding, such as SHVC or MV-HEVC, adiscardable picture may be defined as a coded picture is not used as areference picture for inter prediction and is not used as an inter-layerreference picture in the decoding process of subsequent pictures indecoding order. A discardable picture may be indicated e.g. by a syntaxelement discardable_flag, which may be included e.g. in a slice segmentheader. discardable_flag equal to 1 specifies that the current codedpicture is a discardable picture. discardable_flag equal to 0 specifiesthat the coded picture may be used as a reference picture for interprediction and may be used as an inter-layer reference picture in thedecoding process of subsequent pictures in decoding order. It may bespecified that, when not present, the value of discardable_flag isinferred to be equal to 0.

In scalable and/or multiview coding, such as SHVC and MV-HEVC, a picturein a reference layer that need to be decoded even if the reference layeris not among output layers may be referred to as a key picture. Apicture in an output layer may for example use a key picture as areference for inter-layer prediction, or another picture in thereference layer is used as a reference for inter-layer prediction of apicture in an output layer and the key picture may be used as areference for that another picture. Pictures in a reference layer thatare not key pictures may be referred to as non-key pictures.

Non-key pictures may be classified or characterized for example asfollows:

-   -   A non-key picture that is not used versus a non-key picture that        may be used for sample prediction of other base-layer (BL)        non-key pictures. (If a non-key picture is not used for sample        prediction of other BL pictures, it need not be maintained in        the DPB if the base layer is not among output layers. If a        non-key picture is used for sample prediction of other BL        pictures, in a single-loop decoding operation it needs to be        replaced by the respective decoded EL picture, which introduces        drift.)    -   A non-key picture that is not used versus a non-key picture that        may be used for motion prediction of BL key pictures and/or        other BL non-key pictures.    -   A non-key picture that is not used versus a non-key picture that        may be used for inter-layer sample prediction.    -   A non-key picture that is not used versus a non-key picture that        may be used for inter-layer motion prediction.

An encoder may indicate in the bitstream whether a picture is a keypicture or a non-key picture and/or it may indicate for a non-keypicture one or more of the above-mentioned characteristics. Somedetailed examples of indicating prediction characteristics follow.

In a first example, 2-bit fixed-length coded syntax element, i.e. u(2),here referred to pic_ref_idc, is included in the syntax for a picture,for example in the slice segment header syntax structure, for example bytaking certain two bit positions of the slice_reserved[i] syntax elementof HEVC slice segment header into use. The semantics of pic_ref_idc maybe specified for example as follows:

-   -   pic_ref_idc equal to 3 indicates a discardable picture and        specifies that the coded picture is not used as a reference        picture for inter prediction and is not used as an inter-layer        reference picture in the decoding process of subsequent pictures        in decoding order    -   pic_ref_idc equal to 2 indicates an intra-layer non-reference        picture and specifies that the picture is not used as a        reference picture for inter prediction of subsequent pictures in        decoding order within the same layer and that the picture may be        used as an inter-layer (sample or motion) prediction reference.    -   pic_ref_idc equal to 1 indicates a non-output-layer skip (NOLS)        picture and specifies that the picture may be used as a        reference for inter prediction of subsequent pictures in        decoding order that have pic_ref_idc equal to 1 or 3 until the        next picture in decoding order with pic_ref_idc equal to 0 and        the same or lower TemporalId value than that of the current NOLS        picture. pic_ref_idc equal to 1 also specifies that the picture        is not used as a reference for inter-layer prediction and not        used as a reference for inter prediction of any picture with        pic_ref_idc equal to 0.    -   pic_ref_idc equal to 0 indicates a key picture and specifies        that the picture may be used as a reference for inter prediction        of subsequent pictures in decoding order within the same layer        and may be used as a reference for inter-layer prediction.

A picture marked as intra-layer non-reference picture and either notused or no longer used as a reference picture for inter-layer predictionmay be marked as “unused for reference”.

The signaling of the first example or any similar signaling enabling theidentification of NOLS pictures enables removal of NOLS pictures fromlayers that are not among output layers. It may be required that eitherall discardable pictures and NOLS pictures (of a certain layer) areremoved between two key pictures (of the same certain layer) or that noNOLS pictures between two key pictures (of the same layer) are removed.The removal of NOLS pictures may done by different entities includingbut not limited to the following:

-   -   The removal may be done by a decoder or a bitstream        pre-processor connected to a decoder, which have knowledge on        the output layer set under which the decoding process operates.        Consequently, the output layers can be concluded from the output        layer set in use and NOLS pictures from layers that are not        output but which are among the decoded layers can be removed.        Such removal of NOLS pictures reduces decoding complexity and        the memory required for decoded picture buffering.    -   The removal may be done by an entity modifying the bitstream,        such as a media-aware network element (MANE). The NOLS pictures        may be removed from such layers that are not output layers among        any specified output layer sets. The entity may modify the        indicated output layer sets, for example based on receivers'        properties and/or mode requests, to exclude certain output layer        sets originally present in the indications, e.g. in VPS. For        example, those output layer sets that are concluded to be unused        among receivers or are concluded to be suboptimal for receivers'        properties and/or mode requests may be removed. Consequently,        the number of layers from which NOLS pictures can be removed may        be increased.

A second example is otherwise the same as the first example, but it isspecified that NOLS pictures may be used for inter-layer motionprediction but are not used for inter-layer sample prediction.Consequently, only the picture motion field of the NOLS pictures need tobe decoded when the layer containing the NOLS pictures is not a targetoutput layer, while the sample arrays of these NOLS pictures need not bedecoded or need not be maintained in the DPB.

In a third example, NOLS pictures of both the first and second exampleare separately indicated. For example, in the 2-bit pic_ref_idc syntaxelement, a value indicating an intra-layer non-reference picture may bereplaced by one of the NOLS picture types (of either example 1 orexample 2).

In a fourth example, the NOLS picture is otherwise specified as in thefirst, second or third example, but the semantics of NOLS pictures areindependent of the TemporalId value.

A HRD for a scalable video bitstream may operate similarly to a HRD fora single-layer bitstream. However, some changes may be required ordesirable, particularly when it comes to the DPB operation in multi-loopdecoding of a scalable bitstream. It is possible to specify DPBoperation for multi-loop decoding of a scalable bitstream in multipleways. In a layer-wise approach, each layer may have conceptually its ownDPB, which may otherwise operate independently but some DPB parametersmay be provided jointly for all the layer-wise DPBs and picture outputmay operate synchronously so that the pictures having the same outputtime are output at the same time or, in output order conformancechecking, pictures from the same access unit are output next to eachother. In another approach, referred to as the resolution-specificapproach, layers having the same key properties share the same sub-DPB.The key properties may include one or more of the following: picturewidth, picture height, chroma format, bitdepth, color format/gamut.

It may be possible to support both layer-wise and resolution-specificDPB approach with the same DPB model, which may be referred to as thesub-DPB model. The DPB is partitioned into several sub-DPBs, and eachsub-DPB is otherwise managed independently but some DPB parameters maybe provided jointly for all the sub-DPBs and picture output may operatesynchronously so that the pictures having the same output time areoutput at the same time or, in output order conformance checking,pictures from the same access unit are output next to each other.

A coding standard may include a sub-bitstream extraction process, andsuch is specified for example in SVC, MVC, and HEVC. The sub-bitstreamextraction process relates to converting a bitstream, typically byremoving NAL units, to a sub-bitstream, which may also be referred to asa bitstream subset. The sub-bitstream still remains conforming to thestandard. For example, in HEVC, the bitstream created by excluding allVCL NAL units having a TemporalId value greater than a selected valueand including all other VCL NAL units remains conforming. In HEVC, thesub-bitstream extraction process takes a TemporalId and/or a list ofnuh_layer_id values as input and derives a sub-bitstream (also known asa bitstream subset) by removing from the bitstream all NAL units withTemporalId greater than the input TemporalId value or nuh_layer_id valuenot among the values in the input list of nuh_layer_id values.

A coding standard or system may refer to a term operation point oralike, which may indicate the scalable layers and/or sub-layers underwhich the decoding operates and/or may be associated with asub-bitstream that includes the scalable layers and/or sub-layers beingdecoded. Some non-limiting definitions of an operation point areprovided in the following.

In HEVC, an operation point is defined as bitstream created from anotherbitstream by operation of the sub-bitstream extraction process with theanother bitstream, a target highest TemporalId, and a target layeridentifier list as inputs.

In SHVC and MV-HEVC, an operation point definition may include aconsideration a target output layer set. In SHVC and MV-HEVC, anoperation point may be defined as a bitstream that is created fromanother bitstream by operation of the sub-bitstream extraction processwith the another bitstream, a target highest TemporalId, and a targetlayer identifier list as inputs, and that is associated with a set oftarget output layers.

In MVC, an operation point may be defined as follows: An operation pointis identified by a temporal_id value representing the target temporallevel and a set of view_id values representing the target output views.One operation point is associated with a bitstream subset, whichconsists of the target output views and all other views the targetoutput views depend on, that is derived using the sub-bitstreamextraction process with tIdTarget equal to the temporal_id value andviewIdTargetList consisting of the set of view_id values as inputs. Morethan one operation point may be associated with the same bitstreamsubset. When “an operation point is decoded”, a bitstream subsetcorresponding to the operation point may be decoded and subsequently thetarget output views may be output.

When a bitstream, such as an HEVC bitstream, starts at a CRA or BLApicture, it is not possible to decode the RASL pictures associated withthe CRA or BLA picture correctly, because some reference pictures ofthese RASL pictures might not have been decoded. These RASL pictures aretherefore not output by the decoding process and/or HRD. It may also bepossible to provide external means to impact the decoding process, suchas an interface or an API to the decoder, through which the decoder canbe controlled to treat a CRA picture similarly to a BLA picture or a CRApicture initiating a bitstream and hence omit the output of theassociated RASL pictures. The decoding process may for example associatea variable NoRaslOutputFlag with each IRAP picture and derive a valuefor the variable for example as follows:

-   -   If the current picture is an IDR picture, a BLA picture, the        first picture in the bitstream in decoding order, or the first        picture that follows an end of sequence NAL unit in decoding        order, the variable NoRaslOutputFlag is set equal to 1.    -   Otherwise, if some external means are available to set the        variable HandleCraAsBlaFlag to a value for the current picture,        the variable HandleCraAsBlaFlag is set equal to the value        provided by the external means and the variable NoRaslOutputFlag        is set equal to HandleCraAsBlaFlag.    -   Otherwise, the variable HandleCraAsBlaFlag is set equal to 0 and        the variable NoRaslOutputFlag is set equal to 0.

Available media file format standards include ISO base media file format(ISO/IEC 14496-12, which may be abbreviated ISOBMFF), MPEG-4 file format(ISO/IEC 14496-14, also known as the MP4 format), file format for NALunit structured video (ISO/IEC 14496-15) and 3GPP file format (3GPP TS26.244, also known as the 3GP format). The SVC and MVC file formats arespecified as amendments to the AVC file format. The ISO file format isthe base for derivation of all the above mentioned file formats(excluding the ISO file format itself). These file formats (includingthe ISO file format itself) are generally called the ISO family of fileformats.

The basic building block in the ISO base media file format is called abox. Each box has a header and a payload. The box header indicates thetype of the box and the size of the box in terms of bytes. A box mayenclose other boxes, and the ISO file format specifies which box typesare allowed within a box of a certain type. Furthermore, the presence ofsome boxes may be mandatory in each file, while the presence of otherboxes may be optional. Additionally, for some box types, it may beallowable to have more than one box present in a file. Thus, the ISObase media file format may be considered to specify a hierarchicalstructure of boxes.

According to the ISO family of file formats, a file includes media dataand metadata that are enclosed in separate boxes. In an exampleembodiment, the media data may be provided in a media data (mdat) boxand the movie (moov) box may be used to enclose the metadata. In somecases, for a file to be operable, both of the mdat and moov boxes mustbe present. The movie (moov) box may include one or more tracks, andeach track may reside in one corresponding track box. A track may be oneof the following types: media, hint, timed metadata. A media trackrefers to samples formatted according to a media compression format (andits encapsulation to the ISO base media file format). A hint trackrefers to hint samples, containing cookbook instructions forconstructing packets for transmission over an indicated communicationprotocol. The cookbook instructions may include guidance for packetheader construction and include packet payload construction. In thepacket payload construction, data residing in other tracks or items maybe referenced. As such, for example, data residing in other tracks oritems may be indicated by a reference as to which piece of data in aparticular track or item is instructed to be copied into a packet duringthe packet construction process. A timed metadata track may refer tosamples describing referred media and/or hint samples. For thepresentation of one media type, typically one media track is selected.Samples of a track may be implicitly associated with sample numbers thatare incremented by 1 in the indicated decoding order of samples. Thefirst sample in a track may be associated with sample number 1.

An example of a simplified file structure according to the ISO basemedia file format may be described as follows. The file may include themoov box and the mdat box and the moov box may include one or moretracks that correspond to video and audio, respectively.

The ISO base media file format does not limit a presentation to becontained in one file. As such, a presentation may be comprised withinseveral files. As an example, one file may include the metadata for thewhole presentation and may thereby include all the media data to makethe presentation self-contained. Other files, if used, may not berequired to be formatted to ISO base media file format, and may be usedto include media data, and may also include unused media data, or otherinformation. The ISO base media file format concerns the structure ofthe presentation file only. The format of the media-data files may beconstrained by the ISO base media file format or its derivative formatsonly in that the media-data in the media files is formatted as specifiedin the ISO base media file format or its derivative formats.

The ability to refer to external files may be realized through datareferences. In some examples, a sample description box included in eachtrack may provide a list of sample entries, each providing detailedinformation about the coding type used, and any initializationinformation needed for that coding. All samples of a chunk and allsamples of a track fragment may use the same sample entry. A chunk maybe defined as a contiguous set of samples for one track. The DataReference (dref) box, also included in each track, may define an indexedlist of uniform resource locators (URLs), uniform resource names (URNs),and/or self-references to the file containing the metadata. A sampleentry may point to one index of the Data Reference box, therebyindicating the file containing the samples of the respective chunk ortrack fragment.

Movie fragments may be used when recording content to ISO files in orderto avoid losing data if a recording application crashes, runs out ofmemory space, or some other incident occurs. Without movie fragments,data loss may occur because the file format may typically require thatall metadata, e.g., the movie box, be written in one contiguous area ofthe file. Furthermore, when recording a file, there may not besufficient amount of memory space (e.g., RAM) to buffer a movie box forthe size of the storage available, and re-computing the contents of amovie box when the movie is closed may be too slow. Moreover, moviefragments may enable simultaneous recording and playback of a file usinga regular ISO file parser. Finally, a smaller duration of initialbuffering may be required for progressive downloading, e.g.,simultaneous reception and playback of a file, when movie fragments areused and the initial movie box is smaller compared to a file with thesame media content but structured without movie fragments.

The movie fragment feature may enable splitting the metadata thatconventionally would reside in the movie box into multiple pieces. Eachpiece may correspond to a certain period of time for a track. In otherwords, the movie fragment feature may enable interleaving file metadataand media data. Consequently, the size of the movie box may be limitedand the use cases mentioned above be realized.

In some examples, the media samples for the movie fragments may residein an mdat box, as usual, if they are in the same file as the moov box.For the metadata of the movie fragments, however, a moof box may beprovided. The moof box may include the information for a certainduration of playback time that would previously have been in the moovbox. The moov box may still represent a valid movie on its own, but inaddition, it may include an mvex box indicating that movie fragmentswill follow in the same file. The movie fragments may extend thepresentation that is associated to the moov box in time.

Within the movie fragment there may be a set of track fragments,including anywhere from zero to a plurality per track. The trackfragments may in turn include anywhere from zero to a plurality of trackruns, each of which document is a contiguous run of samples for thattrack. Within these structures, many fields are optional and can bedefaulted. The metadata that may be included in the moof box may belimited to a subset of the metadata that may be included in a moov boxand may be coded differently in some cases. Details regarding the boxesthat can be included in a moof box may be found from the ISO base mediafile format specification.

ISO/IEC 14496-15 specifies Aggregators and Extractors are file formatinternal structures enabling efficient grouping of NAL units orextraction of NAL units from other tracks. While presently ISO/IEC14496-15 specifies the use of Aggregators and Extractors for AVCextensions, such as SVC and MVC, similar NAL-unit-like structures couldbe applied for HEVC extensions too. Aggregators and Extractors use theNAL unit syntax, but their payload has not necessarily been subject tostart code emulation prevention. These structures are seen as NAL unitsin the context of the sample structure of the file format. Whileaccessing a sample, Aggregators must be removed (leaving their containedor referenced NAL Units) and Extractors must be replaced by the datathey reference. Aggregators and Extractors must not be present in astream outside the file format.

ISO/IEC 14496-15 supports aggregation of multiple NAL units into oneaggregator NAL unit (the NAL unit type value of which was taken from thevalue range that is unspecified in the H.264/AVC standard). AggregatorNAL units can both aggregate by inclusion NAL units within them (withinthe size indicated by their length) and also aggregate by reference NALunits that follow them (within the area indicated by theadditional_bytes field within them). When the stream is scanned by anAVC file reader, only the included NAL units are seen as “within” theaggregator. This permits, for example, an AVC file reader to skip awhole set of unneeded SVC or MVC NAL units. SVC NAL units refer to theSVC specific NAL units for which the NAL unit type values are reservedby the AVC specification. MVC NAL units refer to the MVC specific NALunits for which the NAL unit type values are reserved by the AVCspecification. Similarly, if AVC NAL units are aggregated by reference,the AVC reader will not skip them and they remain in-stream for thatreader.

Another benefit achieved through using aggregators is presented in thisparagraph. H.264/AVC, HEVC and their extensions allow an access unit tobe coded in multiple NAL units. The number of NAL units can vary. Inorder to address a dependency representation (of SVC), a layerrepresentation (of SVC) or a coded view (of MVC) as one logical unit byan ISOMBFF parser, one or more aggregator NAL units can be used.Aggregators help in organizing file format samples (e.g. access units)in constant patterns of number of NAL units per a logical scalabilityunit. For example, if all base layer NAL units of an access unit areaggregated into one aggregator, it can be considered that the base layerconsists of one NAL unit. Some of the scalability and/or multiviewproperties are indicated in SVC and MVC file formats through a conceptcalled tiers, which are specified through the sample grouping mechanismof ISOBMFF. Each Scalable Group Entry or Multiview Group Entry, includedas a sample group description entry, documents a subset of the SVCstream or the MVC stream, respectively. Each of the subsets isassociated with a tier and may contain one or more operating points. Thenumber of Scalable Group Entry or Multiview Group Entry is equal to thenumber of different NAL unit sequence pattern to tier assignments. Forexample, one sample group description may indicate that a file formatsample consists of one NAL unit belonging to a first tier and anotherNAL unit belonging to a second tier. Another sample group descriptionentry would be needed for a file format sample consisting of two NALunits belonging to the first tier followed by a NAL unit belonging tothe second tier. However, if an aggregator is used to aggregate the twoNAL units belonging to the first tier, they are considered as one NALunit in the sample group description entries. Consequently, aggregatorshelp in reducing the amount of signaling needed to specify tiers.

Two forms of aggregation are supported currently by the aggregators. Inthe first, all samples that are aggregated are included within theaggregator NAL unit themselves. This form of aggregation is calledaggregation by inclusion. In the second form, all aggregated sampleswhich are always stored as continued bytes in the ISOMBFF storage formatare signaled by reference. The aggregated NAL units are not containedwithin the Aggregator NAL unit but are referred to from within theAggregator NAL unit. The syntax of the aggregator NAL unit includes anadditional_bytes field, which indicates the number of bytes aggregatedby reference. It is possible to have a single aggregator that aggregatesboth by inclusion and by reference by defining the length or size of theaggregator to include NAL units and by defining a non-zeroadditional_bytes value to aggregate NAL units by reference.

HEVC enable encoders and/or other entities, such as a splicers, tocontrol picture output from the decoding process and/or HRD with thevalues of the pic_output_flag and/or no_output_of prior_pics_flag syntaxelements as described in the following.

The pic_output_flag syntax element, which may be present in the slicesegment header, affects the derivation of the variable PicOutputFlag.Each decoded picture is associated with a value of PicOutputFlag (whichmay differ from the value of PicOutputFlag of other pictures).PicOutputFlag is set to 1 for pictures that are not RASL picturesassociated with an IRAP picture with NoRaslOutputFlag equal to 1.Otherwise, PicOutputFlag is set to be equal to pic_output_flag. Pictureswith PicOutputFlag equal to 0 are not output. Picture with PicOutputFlagequal to 1 are output, unless they become affected byNoOutputOfPriorPicsFlag, as explained in the following.

The no_output_of prior_pics_flag syntax element, which is present forIRAP pictures in the slice segment header, affects the output ofpreviously-decoded pictures in the decoded picture buffer after thedecoding of an IDR or a BLA picture that is not the first picture in thebitstream as follows: The removal of pictures from the DPB beforedecoding of the current picture (but after parsing the slice header ofthe first slice of the current picture) happens instantaneously at theCPB removal time of the first decoding unit of access unit n (containingthe current picture) and proceeds as follows:

-   -   The reference picture set of the current picture is decoded.    -   When the current picture is an IRAP picture with        NoRaslOutputFlag equal to 1 that is not picture 0, the following        ordered steps are applied:        -   The variable NoOutputOfPriorPicsFlag is derived as follows:            -   If the current picture is a CRA picture,                NoOutputOfPriorPicsFlag is set equal to 1 (regardless of                the value of no_output_of prior_pics_flag).            -   Otherwise, if the width, height, or the DPB capacity of                the current picture differ from those of the previous                picture, NoOutputOfPriorPicsFlag may but need not be set                equal to 1 or may be set equal to                no_output_of_prior_pics_flag.            -   Otherwise, NoOutputOfPriorPicsFlag is set equal to                no_output_of prior_pics_flag.        -   The value of NoOutputOfPriorPicsFlag derived for the decoder            under test is applied for the HRD, such that when the value            of NoOutputOfPriorPicsFlag is equal to 1, all picture            storage buffers in the DPB are emptied without output of the            pictures they contain, and the DPB fullness is set equal to            0.    -   When both of the following conditions are true for any pictures        k in the DPB, all such pictures k in the DPB are removed from        the DPB:        -   Picture k is marked as “unused for reference”.        -   Picture k has PicOutputFlag equal to 0 or its DPB output            time is less than or equal to the CPB removal time of the            first decoding unit (denoted as decoding unit m) of the            current picture n.    -   For each picture that is removed from the DPB, the DPB fullness        is decremented by one.

Splicers or other entities concatenating coded video sequences orbitstreams may set no_output_of_prior_pics_flag equal to 1 for examplewhen the output of a spliced coded video sequence might overlap with theoutput of some picture in the previous coded video sequence.

A coded video sequence (CVS) in scalable extensions of HEVC may bespecified as follows: A coded video sequence is a sequence of accessunits that consists, in decoding order, of an TRAP access unitcontaining an IRAP picture having nuh_layer_id equal to 0 andNoRaslOutputFlag equal to 1, followed by zero or more access units thatare not IRAP access units containing an IRAP picture having nuh_layer_idequal to 0 and NoRaslOutputFlag equal to 1, including all subsequentaccess units up to but not including any subsequent access unit that isan IRAP access unit containing an IRAP picture having nuh_layer_id equalto 0 and NoRaslOutputFlag equal to 1.

Referring back to the POC coding and derivation in H.265/HEVC, thefollowing design decisions that may affect the POC derivation or codinghave been made for H.265/HEVC scalable and/or multiview extensions:

-   -   IRAP pictures need not be aligned. In other words, when one        layer has an IRAP picture in an AU, it is not required that the        other layers have an IRAP picture.    -   An access unit needs not contain pictures in all layers (which        are represented by the bitstream).    -   The bitstream may contain independently coded layers (other than        the base layer) that do not use inter-layer prediction from any        other layer. For example, an auxiliary picture layer may be        independent of the base layer.

As a consequence the following features or use cases are enabled.

Adaptive layer up-switching and down-switching refers to a process wherea middle-box and/or a decoder may determine at any access unit that thehighest layer is no longer forwarded and/or decoded, respectively (or,more generally, how many layers are forwarded and/or decoded,respectively)—this may be referred to as layer down-switching. Likewise,a middle-box and/or a decoder may determine to start forwarding and/ordecoding, respectively, an additional layer (the next higher than thehighest layer forwarded and/or decoded, respectively, before) and thatadditional layer has an IRAP picture—this may be referred to as layerup-switching.

Adaptive Resolution Change (ARC) refers to dynamically changing theresolution within the video sequence, for example in video-conferencinguse-cases. Adaptive Resolution Change may be used e.g. for betternetwork adaptation and error resilience. For better adaptation tochanging network requirements for different content, it may be desiredto be able to change both the temporal/spatial resolution in addition toquality. The Adaptive Resolution Change may also enable a fast start,wherein the start-up time of a session may be able to be increased byfirst sending a low resolution frame and then increasing the resolution.The Adaptive Resolution Change may further be used in composing aconference. For example, when a person starts speaking, his/hercorresponding resolution may be increased. Doing this with an IDR framemay cause a “blip” in the quality as IDR frames need to be coded at arelatively low quality so that the delay is not significantly increased.

In the following some details of an adaptive resolution change use-casesare described in more detail using the scalable video coding framework.As scalable video coding inherently includes mechanisms for resolutionchange, the adaptive resolution change could efficiently be supported asindicated in FIG. 4.

In the example of FIG. 4, switching happens at picture 3 (BL3) and thedecoder receives the bitstream with following pictures:BL0-BL1-BL2-BL3-EL3-EL4-EL6-EL6 . . . Pictures BL0-BL3 are pictures of abase layer and pictures EL3-EL6 . . . are pictures of an enhancementlayer. In this example pictures BL0-BL3 and EL4-EL6 use only intra-layerprediction and the picture EL3 uses either or both intra prediction andinter-layer prediction.

A sequence level signaling may be used to indicate the decoder if thereis a resolution change in the bitstream. This may be performed e.g. byusing a flag single_layer_for_non_irap_flag. Thesingle_layer_for_non_irap_flag may be used to signal that a coded videosequence is constrained to perform the adaptive resolution changeoperation. The single_layer_for_non_irap_flag specifically indicatesthat, except for switching pictures, each AU in the sequence contains asingle picture from a single layer (which may or may not be BL picture);and that access units where switching happens include pictures from twolayers, the EL picture in such an access unit is an IRAP picture, andinter-layer prediction may be used for the EL IRAP picture.

Indicating single_layer_for_non_irap_flag in VPS allows the decoder toknow that scalability is not used except for achieving resolutionchange, so that the decoder resources can be allocated accordingly upona session start.

Scalable bitstreams with IRAP pictures or similar that are not alignedacross layers may be used for example more frequent IRAP pictures can beused in the base layer, where they may have a smaller coded size due toe.g. a smaller spatial resolution. A process or mechanism for layer-wisestart-up of the decoding may be included in a video decoding scheme.Decoders may hence start decoding of a bitstream when a base layercontains an IRAP picture and step-wise start decoding other layers whenthey contain IRAP pictures. In other words, in a layer-wise start-up ofthe decoding process, decoders progressively increase the number ofdecoded layers (where layers may represent an enhancement in spatialresolution, quality level, views, additional components such as depth,or a combination) as subsequent pictures from additional enhancementlayers are decoded in the decoding process. The progressive increase ofthe number of decoded layers may be perceived for example as aprogressive improvement of picture quality (in case of quality andspatial scalability).

A layer-wise start-up mechanism may generate unavailable pictures forthe reference pictures of the first picture in decoding order in aparticular enhancement layer. Alternatively, a decoder may omit thedecoding of pictures preceding the IRAP picture from which the decodingof a layer can be started. These pictures that may be omitted may bespecifically labeled by the encoder or another entity within thebitstream. For example, one or more specific NAL unit types may be usedfor them. These pictures may be referred to as cross-layer random accessskip (CL-RAS) pictures.

A layer-wise start-up mechanism may start the output of enhancementlayer pictures from an IRAP picture in that enhancement layer, when allreference layers of that enhancement layer have been initializedsimilarly with an IRAP picture in the reference layers. In other words,any pictures (within the same layer) preceding such an IRAP picture inoutput order might not be output from the decoder and/or might not bedisplayed. In some cases, decodable leading pictures associated withsuch an IRAP picture may be output while other pictures preceding suchan IRAP picture might not be output.

Concatenation of coded video data, which may also be referred to assplicing, may occur for example coded video sequences are concatenatedinto a bitstream that is broadcast or streamed or stored in a massmemory. For example, coded video sequences representing commercials oradvertisements may be concatenated with movies or other “primary”content.

Scalable video bitstreams might contain IRAP pictures that are notaligned across layers. It may, however, be convenient to enableconcatenation of a coded video sequence that contains an IRAP picture inthe base layer in its first access unit but not necessarily in alllayers. A second coded video sequence that is spliced after a firstcoded video sequence should trigger a layer-wise decoding start-upprocess. That is because the first access unit of said second codedvideo sequence might not contain an IRAP picture in all its layers andhence some reference pictures for the non-IRAP pictures in that accessunit may not be available (in the concatenated bitstream) and cannottherefore be decoded. The entity concatenating the coded videosequences, hereafter referred to as the splicer, should therefore modifythe first access unit of the second coded video sequence such that ittriggers a layer-wise start-up process in decoder(s).

Indication(s) may exist in the bitstream syntax to indicate triggeringof a layer-wise start-up process. These indication(s) may be generatedby encoders or splicers and may be obeyed by decoders. Theseindication(s) may be used for particular picture type(s) or NAL unittype(s) only, such as only for IDR pictures, while in other embodimentsthese indication(s) may be used for any picture type(s). Without loss ofgenerality, an indication called cross_layer_bla_flag that is consideredto be included in a slice segment header is referred to below. It shouldbe understood that a similar indication with any other name or includedin any other syntax structures could be additionally or alternativelyused.

Independently of indication(s) triggering a layer-wise start-up process,certain NAL unit type(s) and/or picture type(s) may trigger a layer-wisestart-up process. For example, a base-layer BLA picture may trigger alayer-wise start-up process.

A layer-wise start-up mechanism may be initiated in one or more of thefollowing cases:

-   -   At the beginning of a bitstream.    -   At the beginning of a coded video sequence, when specifically        controlled, e.g. when a decoding process is started or        re-started e.g. as response to tuning into a broadcast or        seeking to a position in a file or stream. The decoding process        may input an variable, e.g. referred to as NoClrasOutputFlag,        that may be controlled by external means, such as the video        player or alike.    -   A base-layer BLA picture.    -   A base-layer IDR picture with cross_layer_bla_flag equal to 1.        (Or a base-layer IRAP picture with cross_layer_bla_flag equal to        1.)

Cross-layer random access skipped (CL-RAS) pictures may have theproperty that when a layer-wise start-up mechanism is invoked (e.g. whenNoClrasOutputFlag is equal to 1), the CL-RAS pictures are not output andmay not be correctly decodable, as the CL-RAS picture may containreferences to pictures that are not present in the bitstream. It may bespecified that CL-RAS pictures are not used as reference pictures forthe decoding process of non-CL-RAS pictures.

CL-RAS pictures may be explicitly indicated e.g. by one or more NAL unittypes or slice header flags (e.g. by re-naming cross_layer_bla_flag tocross_layer_constraint_flag and re-defining the semantics ofcross_layer_bla_flag for non-IRAP pictures). A picture may be consideredas a CL-RAS picture when it is a non-IRAP picture (e.g. as determined byits NAL unit type), it resides in an enhancement layer and it hascross_layer_constraint_flag (or similar) equal to 1. Otherwise, apicture may be classified of being a non-CL-RAS picture.cross_layer_bla_flag may be inferred to be equal to 1 (or a respectivevariable may be set to 1), if the picture is an IRAP picture (e.g. asdetermined by its NAL unit type), it resides in the base layer, andcross_layer_constraint_flag is equal to 1. Otherwise,cross_layer_bla_flag may inferred to be equal to 0 (or a respectivevariable may be set to 0). Alternatively, CL-RAS pictures may beinferred. For example, a picture with nuh_layer_id equal to layerId maybe inferred to be a CL-RAS picture when theLayerInitializedFlag[layerId] is equal to 0.

A decoding process may be specified in a manner that a certain variablecontrols whether or not a layer-wise start-up process is used. Forexample, a variable NoClrasOutputFlag may be used, which, when equal to0, indicates a normal decoding operation, and when equal to 1, indicatesa layer-wise start-up operation. NoClrasOutputFlag may be set forexample using one or more of the following steps:

1) If the current picture is an IRAP picture that is the first picturein the bitstream, NoClrasOutputFlag is set equal to 1.

2) Otherwise, if some external means are available to set the variableNoClrasOutputFlag equal to a value for a base-layer IRAP picture, thevariable NoClrasOutputFlag is set equal to the value provided by theexternal means.

3) Otherwise, if the current picture is a BLA picture that is the firstpicture in a coded video sequence (CVS), NoClrasOutputFlag is set equalto 1.

4) Otherwise, if the current picture is an IDR picture that is the firstpicture in a coded video sequence (CVS) and cross_layer_bla_flag isequal to 1, NoClrasOutputFlag is set equal to 1.

5) Otherwise, NoClrasOutputFlag is set equal to 0.

Step 4 above may alternatively be phrased more generally for example asfollows: “Otherwise, if the current picture is an IRAP picture that isthe first picture in a CVS and an indication of layer-wise start-upprocess is associated with the IRAP picture, NoClrasOutputFlag is setequal to 1.” Step 3 above may be removed, and the BLA picture may bespecified to initiate a layer-wise start-up process (i.e. setNoClrasOutputFlag equal to 1), when cross_layer_bla_flag for it is equalto 1. It should be understood that other ways to phrase the conditionare possible and equally applicable.

A decoding process for layer-wise start-up may be for example controlledby two array variables LayerInitializedFlag[i] andFirstPicInLayerDecodedFlag[i] which may have entries for each layer(possibly excluding the base layer and possibly other independent layerstoo). When the layer-wise start-up process is invoked, for example asresponse to NoClrasOutputFlag being equal to 1, these array variablesmay be reset to their default values. For example, when there 64 layersare enabled (e.g. with a 6-bit nuh_layer_id), the variables may be resetas follows: the variable LayerInitializedFlag[i] is set equal to 0 forall values of i from 0 to 63, inclusive, and the variableFirstPicInLayerDecodedFlag[i] is set equal to 0 for all values of i from1 to 63, inclusive.

The decoding process may include the following or similar to control theoutput of RASL pictures. When the current picture is an IRAP picture,the following applies:

-   -   If LayerInitializedFlag[nuh_layer_id] is equal to 0, the        variable NoRaslOutputFlag is set equal to 1.    -   Otherwise, if some external means is available to set the        variable HandleCraAsBlaFlag to a value for the current picture,        the variable HandleCraAsBlaFlag is set equal to the value        provided by the external means and the variable NoRaslOutputFlag        is set equal to HandleCraAsBlaFlag.    -   Otherwise, the variable HandleCraAsBlaFlag is set equal to 0 and        the variable NoRaslOutputFlag is set equal to 0.

The decoding process may include the following to update theLayerInitializedFlag for a layer.

When the current picture is an IRAP picture and either one of thefollowing is true, LayerInitializedFlag[nuh_layer_id] is set equal to 1.

-   -   nuh_layer_id is equal to 0.    -   LayerInitializedFlag[nuh_layer_id] is equal to 0 and        LayerInitializedFlag[refLayerId] is equal to 1 for all values of        refLayerId equal to RefLayerId[nuh_layer_id][j], where j is in        the range of 0 to NumDirectRefLayers[nuh_layer_id]−1, inclusive.

The encoder process may be constrained so that it is required that whenthe current picture is an IRAP picture, either one of the following istrue:

-   -   nuh_layer_id is equal to 0.    -   LayerInitializedFlag[refLayerId] is equal to 1 for all values of        refLayerId equal to RefLayerId[nuh_layer_id][j], where j is in        the range of 0 to NumDirectRefLayers[nuh_layer_id]−1, inclusive.

The above-mentioned constraint in the encoding process may bepre-defined for example in a coding standard, or the above-mentionedconstraint may be indicated by the encoder in the bitstream. If theabove-mentioned constraint is pre-defined or decoded by from thebitstream to be followed in a bitstream or a part thereof, the decodingprocess may set LayerInitializedFlag[nuh_layer_id] equal to 1 when thecurrent picture is an IRAP picture.

When FirstPicInLayerDecodedFlag[nuh_layer_id] is equal to 0, thedecoding process for generating unavailable reference pictures may beinvoked prior to decoding the current picture. The decoding process forgenerating unavailable reference pictures may generate pictures for eachpicture in a reference picture set with default values. The process ofgenerating unavailable reference pictures may be primarily specifiedonly for the specification of syntax constraints for CL-RAS pictures,where a CL-RAS picture may be defined as a picture with nuh_layer_idequal to layerId and LayerInitializedFlag[layerId] is equal to 0. In HRDoperations, CL-RAS pictures may need to be taken into consideration inderivation of CPB arrival and removal times. In some embodiments,decoders may ignore any CL-RAS pictures, as these pictures are notspecified for output and have no effect on the decoding process of anyother pictures that are specified for output.

The MV-HEVC/SHVC draft specification (in documents JCT3V-E1004 andJCTVC-N1008, respectively) derives a default output layer set for eachlayer set specified in the VPS. The VPS extension flagdefault_one_target_output_layer_flag, when equal to 0, specifies thateach layer is an output layer, and when equal to 1, specifies that onlythe highest layer is an output layer. In addition, to the default outputlayer sets, the VPS extension enables to specify additional output layersets with selected layers indicated to be output layers.

While a constant set of output layers suits well use cases andbitstreams where the highest layer stays unchanged in each access unit,they do not support use cases where the highest layer changes from oneaccess unit to another. It has therefore been proposed that each outputlayer in an output layer set is associated with a minimum alternativeoutput layer. The minimum alternative output layer for each output layerof each output layer set is indicated in a sequence-level syntaxstructure (e.g. VPS in H.265/HEVC and its extensions). It is used tospecify the minimum layer identifier value of a picture that can beoutput instead of the specified output layer, if a picture at the outputlayer is not present in an access unit. The first direct or indirectinter-layer reference picture present in the access unit in descendinglayer identifier order down to the indicated minimum alternative outputlayer is output. For the default output layer sets, the minimum outputlayer may be specified to be the same as the output layer; i.e. when adefault output layer set is in use, the decoder outputs only thespecified output layers. For the additional output layer sets, one ormore syntax elements may control the assignment of alternative outputlayers. For example, a VPS extension may include u(1)-codedalt_output_layer_flag. When alt_output_layer_flag is equal to 0,pictures that are not at the target output layers are not output. Whenalt_output_layer_flag equal to 1 and a picture at the a target outputlayer is not present in an access unit, a picture with highestnuh_layer_id among those pictures of the access unit for whichPicOutputFlag is equal to 1 and which are not among the target outputlayers is output. In another example, the syntax elementmin_alt_output_layer_idx[i][j] is added to the VPS extension syntax foreach output layer of the non-default output layer sets and specifies thealternative output layer index for the j-th layer within the i-th outputlayer set.

A POC reset approach, for which the latest proposal is in documentJCTVC-O0213v4 (also published as JCT3V-F0073v4), is based on indicatingwithin a slice header that POC values are to be reset so that the POC ofthe current picture is derived from the provided POC signaling for thecurrent picture and the POCs of the earlier pictures, in decoding order,are decremented by a certain value.

An example of the POC reset approach is given in FIG. 5. In thisexample, there is an IDR picture in the enhancement layer (EL). For allpictures in the access unit containing the EL IDR picture, a POC MSBreset is indicated. This indication has the impact that the POC of thataccess unit will have POC MSB equal to 0 and POC LSB equal to thesignaled value (i.e. pic_order_cnt_lsb). In this example, the POC ofthat AU becomes 10. The POCs of the pictures that are in the DPB aresubtracted by a value, which may be referred to as DeltaPocVal. TheDeltaPocVal is derived on the basis of a previous picture count,PrevPicOrderCnt[nuh_layer_id], which is the PicOderCntVal of theprevious picture that is not a RASL picture, a RADL picture, or asub-layer non-reference picture, has TemporalId equal to 0, and has thesame nuh_layer_id value as the current picture. In the example,PrevPicOrderCnt for the EL IDR picture is 1032. PrevPicOrderCnt is splitinto two parts, prevPicOrderCntLsb and prevPicOrderCntMsb, the sum ofwhich is equal to PrevPicOrderCnt. In the example, MaxPicOrderCntLsb is16, and consequently prevPicOrderCntMsb is 1024 (which is divisible by16) and prevPicOrderLsb is equal to 8. For POC MSB reset the DeltaPocValis set equal to prevPicOrderCntMsb, i.e. 1024 in this example. Theupdated POC values are shown in the FIG. 5 (with the notation: “→updated POC value”).

-   -   Altogether the proposed POC reset signaling also contains four        modes of POC resetting:    -   POC MSB reset in the current access unit. This can be used when        an enhancement layer contains an IRAP picture. (This mode is        indicated in the syntax by poc_reset_idc equal to 1.)    -   Full POC reset (both MSB and LSB to 0) in the current access        unit. This can be used when the base layer contains an IDR        picture. (This mode is indicated in the syntax by poc_reset_idc        equal to 2.)

“Delayed” POC MSB reset. This can be used for a picture of nuh_layer_idequal to nuhLayerId such that there was no picture in of nuh_layer_idequal to nuhLayerId in the earlier access unit (in decoding order) thatcaused a POC MSB reset. (This mode is indicated in the syntax bypoc_reset_idc equal to 3 and full_poc_reset_flag equal to 0.)

“Delayed” full POC reset. This can be used for a picture of nuh_layer_idequal to nuhLayerId such that there was no picture in of nuh_layer_idequal to nuhLayerId in the earlier access unit (in decoding order) thatcaused a full POC reset. (This mode is indicated in the syntax bypoc_reset_idc equal to 3 and full_poc_reset_flag equal to 1.)

The “delayed” POC reset signaling can also be used for error resiliencepurpose (to provide resilience against a loss of a previous picture inthe same layer including the POC reset signaling).

A concept of POC resetting period is specified based on the POCresetting period ID. Each non-IRAP picture that belongs to an accessunit that contains at least one IRAP picture must be the start of a POCresetting period in the layer containing the non-IRAP picture. In thataccess unit, each picture would be the start of a POC resetting periodin the layer containing the picture. POC resetting and update of POCvalues of same-layer pictures in the DPB are applied only for the firstpicture within each POC resetting period.

POC values of earlier pictures of all layers in the DPB are updated atthe beginning of each access unit that requires POC reset and starts anew POC resetting period (before the decoding of the first picturereceived for the access unit but after parsing and decoding of the sliceheader information of the first slice of that picture).

For derivation of the delta POC value used for updating the POC valuesof the same-layer pictures in the DPB as well as for derivation of thePOC MSB of the POC value of the current picture, a POC LSB value(poc_lsb_val syntax element) is conditionally signalled in the slicesegment header (for the “delayed” POC reset modes as well as forbase-layer pictures with full POC reset, such as base-layer IDRpictures). When “delayed” POC reset modes are used, poc_lsb_val may beset equal to the value POC LSB (slice_pic_order_cnt_lsb) of the accessunit in which the POC was reset. When a full POC reset is used in thebase layer, the poc_lsb_val may be set equal to POC LSB of prevTid0Pic(as specified earlier).

The decoding process of deriving PicOrderCntVal, the picture order countof the current picture, is described in more details below.

The function GetCurrMsb is specified as follows.

${{GetCurrMsb}\left( {{cl},{pl},{pm},{ml}} \right)} = \left\{ \begin{matrix}{{{pm} + {ml}};} & {{{pl} - {cl}}>={{ml}/2}} \\{{{pm} - {ml}};} & {{{cl} - {pl}} > {{ml}/2}} \\{{pm};} & {otherwise}\end{matrix} \right.$

If FirstPiclnLayerDecodedFlag[nuh_layer_id] is equal to 1, poc_reset_idcis greater than 0, and the access unit containing the current picture isthe first access unit in decoding order in a POC resetting period, thefollowing applies:

-   -   The variables pocMsbDelta, pocLsbDelta and DeltaPocVal are        derived as follows:

prevPicOrderCntLsb= prevPicOrderCnt[ nuh_layer_id ] & (MaxPicOrderCntLsb − 1 ) prevPicOrderCntMsb = PrevPicOrderCnt[nuh_layer_id ] − prevPicOrderCntLsb if( poc_reset_idc == 3 ∥ (poc_reset_idc == 2 && nuh_layer_id == 0 ) ) pocLsbVal = poc_lsb_val elsepocLsbVal = slice_pic_order_cnt_lsb pocMsbDelta = getCurrMsb( pocLsbVal,prevPicOrderCntLsb, prevPicOrderCntMsb, MaxPicOrderCntLsb ) if(poc_reset_idc == 2 ∥ ( poc_reset_idc == 3 && full_poc_reset_flag ) )pocLsbDelta = pocLsbVal else pocLsbDelta = 0 DeltaPocVal = pocMsbDelta +pocLsbDelta

-   -   The PicOrderCntVal of the current picture is derived as follows:

if( poc_reset_idc == 1 ) PicOrderCntVal = slice_pic_order_cnt_lsb elseif( poc_reset_idc == 2 ) PicOrderCntVal = 0 else { // poc_reset_idc == 3PicOrderCntMsb = getCurrMsb( slice_pic_order_cnt_lsb,full_poc_reset_flag ? 0 : poc_lsb_val, 0, MaxPicOrderCntLsb )PicOrderCntVal = PicOrderCntMsb + slice_pic_order_cnt_lsb }

The value of PrevPicOrderCnt[nuh_layer_id] is derived as follows:

-   -   If the current picture is not a RASL picture, a RADL picture, or        a sub-layer non-reference picture, and the current picture has        TemporalId equal to 0, PrevPicOrderCnt[nuh_layer_id] is set        equal to PicOrderCntVal.    -   Otherwise when poc_reset_idc is equal to 3,        PrevPicOrderCnt[nuh_layer_id] is set equal to        full_poc_reset_flag ? 0: poc_lsb_val.

Otherwise, the following applies:

-   -   The value of DeltaPocVal is set equal to 0.    -   The PicOrderCntVal of the current picture is derived as follows:

if( !FirstPicInLayerDecodedFlag[ nuh_layer_id ] ) { if( poc_reset_idc ==1 ) PicOrderCntVal = slice_pic_order_cnt_lsb else if( poc_reset_idc == 2) PicOrderCntVal = 0 else if( poc_reset_idc == 3 ) { PicOrderCntMsb =getCurrMsb(slice_pic_order_cnt_lsb, full_poc_reset_flag ? 0 :poc_lsb_val, 0, MaxPicOrderCntLsb ) PicOrderCntVal = PicOrderCntMsb +slice_pic_order_cnt_lsb  } else // the current picture is an IRAPpicture with NoRaslOutputFlag   equal to 1 PicOrderCntVal =slice_pic_order_cnt_lsb } else { // the POC derivation as in HEVCversion 1  if( the current picture is an IRAP picture withNoRaslOutputFlag equal  to 1 ) PicOrderCntMsb = 0  else {prevPicOrderCntLsb = PrevPicOrderCnt[ nuh_layer_id ] & (MaxPicOrderCntLsb − 1 ) prevPicOrderCntMsb = PrevPicOrderCnt[nuh_layer_id ] − prevPicOrderCntLsb PicOrderCntMsb = getCurrMsb(slice_pic_order_cnt_lsb, prevPicOrderCntLsb, prevPicOrderCntMsb,MaxPicOrderCntLsb )  }  PicOrderCntVal = PicOrderCntMsb +slice_pic_order_cnt_lsb }

The value of PrevPicOrderCnt[nuh_layer_id] is derived as follows:

-   -   If the current picture is not a RASL picture, a RADL picture, or        a sub-layer non-reference picture, and the current picture has        TemporalId equal to 0, PrevPicOrderCnt[nuh_layer_id] is set        equal to PicOrderCntVal.    -   Otherwise when FirstPicInLayerDecodedFlag[nuh_layer_id] is equal        to 0 and poc_reset_idc is equal to 3,        PrevPicOrderCnt[nuh_layer_id] is set equal to        full_poc_reset_flag ? 0:poc_lsb_val.

If the current picture is the first picture in an access unit (i.e. thepicture with the lowest value of nuh_layer_id among all pictures in theaccess unit) and DeltaPocVal is greater than 0, the PicOrderCntValvalues of all decoded pictures of all layers in the DPB are decrementedby the value of DeltaPocVal.

At this moment in the decoding process, the current picture is not yetdecoded, thus the above operation does not affect the PicOrderCntValvalue of the current picture.

At least the following problems are encountered in the POC resetapproach:

1. An MV-HEVC/SHVC base-layer decoder would conclude different POCvalues than a single-layer (version 1) decoder. This would complicatethe use of an existing HEVC v1 implementation (e.g. HW implementation)for decoding the base layer of a scalable bitstream.

2. The POC values are no longer static. This would complicate the use ofPOC values to label pictures in post-processing operations, which isrequired for example to conclude the persistence of some SEI messages.

-   -   Background: the persistence of the following SEI messages is        specified similarly to the following: [the message pertains        until] “the PicOrderCntVal of the next slice to be decoded is        greater than currPicOrderCntVal+pic_order_cnt_delta, where        currPicOrderCntVal is the value of PicOrderCntVal of the picture        in the access unit containing the SEI message”        -   i. Recovery point SEI message        -   ii. Progressive refinement start SEI message        -   iii. Region refresh information SEI message    -   The structure of pictures information SEI message also has        similar persistence than the SEI messages named above and        additionally uses POC within the persistence scope to identify        pictures.    -   Additionally, pan-scan rectangle, tone mapping information,        frame packing arrangement, and display orientation SEI messages        cease to persist when a picture with a greater POC value than        that of the current picture are output.

3. POC values may be used to label decoded pictures for usages outsidethe context of the decoder and/or the encoder. For example, POC valuesmay be used to indicate pictures in feedback-based error control inwhich a decoder sends information to the far-end encoder on the successand/or failure of decoding of pictures and/or reception and/or loss ofpictures. For example, one or more of the following kinds of feedbackmessages may be generated:

-   -   A certain picture was lost (not received) or was not correctly        decoded.    -   A certain picture was correctly decoded.    -   Encoding of subsequent pictures is desired or commanded to use        only certain picture(s) which have been correctly received.

If POC are dynamically updated based on POC reset information, thedecoder and the far-end encoder have to have the same state of the POCderivation state machine to interpret the POC values of the feedbackmessages correctly. In other words, a POC value alone is not sufficientfor the far-end encoder to determine which picture the decoder pointsat.

The layer-wise POC approach, for which the latest proposal is indocument JCTVC-O0275v3 (also published as JCT3V-F0092v3), creates astatic label for each picture. For the base-layer pictures, the POC asspecified in H.265/HEVC version 1 is used as the static label. Forenhancement-layer pictures, a layer-wise POC is used as the staticlabel. The problems listed above are therefore avoided.

The layer-wise POC proposal includes the following ingredients:

1. Slice headers of enhancement-layer slices conditionally containsignalling for a layer-wise POC value (layer_delta_pic_order_cnt_minus1in versions 1 and 2 of JCTVC-O0275/JCT3V-F0092, layer_pic_order_cnt_lsbin version 3 of JCTVC-O0275/JCT3V-F0092), which is intended to be usedin the first picture of each layer after a base-layer IDR picture. Thepresence of the layer-wise POC signalling in the slice header is gatedby a slice header flag and a PPS flag.

The layer-wise POC LSB value is intended to be included in the firstpicture of an enhancement layer after the base-layer included an IDRpicture. The layer-wise POC LSB value is used to derive the layer-wisePOC difference of the previous reference picture on the same layer withTemporalID equal to 0 and the current picture as if the POC values werenot reset in between them.

2. A layer-wise POC value, LayerPicOrderCntVal is derived for eachenhancement-layer picture based on slice header signalling, if present,and based on PicOrderCntVal differences, otherwise.

3. The function DiffPicOrderCnt( ) is specified to useLayerPicOrderCntVal values.

It is remarked that PicOrderCntVal is used in the decoding process onlyas input to the DiffPicOrderCnt( ) function. This opens the possibilityto re-specify the DiffPicOrderCnt( ) function to use other inputs thanPicOrderCntVal.

4. Reference picture sets for enhancement layers use layer-wise POCvalues.

An example of the layer-wise POC approach is given in FIG. 6. In thisexample an EL IDR picture is presented. Its layer-wise POC is derived onthe basis of the previous picture (prevTid0Pic) that is not a RASLpicture, a RADL picture, or a sub-layer non-reference picture, hasTemporalId equal to 0, and has the same nuh_layer_id value as thecurrent picture. In this case, no additional signalling is required asslice_pic_order_cnt_lsb of the EL IDR picture can be used to derive bothPOC and layer-wise POC.

Another example of the layer-wise POC approach is presented in FIGS. 7aand 7b . In this example, there is a base-layer IDR picture, whichresets the POC values across the same access unit to be equal to 0.However, it is desirable to have the layer-wise POC values derived bycontinuing the same numbering principle as if the POC values were notreset in order to have proper motion vector scaling in the TMVP process,for example. Hence, a layer-wise POC value is indicated for the ELpicture that resides in the same access unit as the base-layer IDRpicture. The signalling may be a difference of layer-wise POC valuesbetween this picture and prevTid0Pic (in FIG. 7a) or an POC LSB valuethat indicates the layer-wise POC MSB and LSB relation to those ofprevTid0Pic (in FIG. 7b ).

Both the POC reset approach and layer-wise POC approach use a previouspicture (prevTid0Pic) that is not a RASL picture, a RADL picture, or asub-layer non-reference picture, has TemporalId equal to 0, and has thesame nuh_layer_id value as the current picture. In both approaches, thePOC of the prevTid0Pic is used to derive the POC of the current picturein the same layer. Additionally, the POC reset approach uses prevTid0Picto determine the delta POC value to decrement from the earlier picturesin decoding order in the DPB, and the layer-wise POC approach usesprevTid0Pic to determine the layer-wise POC value. These design choiceshave the consequences presented in the following paragraphs.

In order to have the POC reset approach work correctly, the prevTid0Picin the layer that triggers the decrementing of the POC values acrosslayers shall be less than (MaxPicOrderCntLsb/2) apart from the POC valueof the current picture. Otherwise, the delta POC value is derivedwrongly. While in conventional scalable picture coding structuresprevTid0Pic typically satisfies this constraint, the following problemshave been found:

1. Especially when layer down-switching has occurred before the POCreset point, the POC reset approach becomes vulnerable to picturelosses. This is illustrated in FIG. 8, in which the same bitstream as inFIG. 5 is used and considered from the viewpoint of a receiver. A layerdown-switching took place after the access unit with POC equal to 1000(which could have been initiated by the sender, by a middle-box, or bythe receiver). The base-layer picture at access unit was lost intransmission. Hence, in the receiver, the EL picture of access unit 10initiates the decrementing of the POC values of the pictures in the DPB.The PrevPicOrderCnt of the EL is 1000, which is more than(MaxPicOrderCntLsb/2) apart from the real POC of the AU with POC 10. Thedecoder has no means to conclude how many times the POC LSBs havewrapped over between prevTid0Pic (with POC equal to PrevPicOrderCnt).Hence, the POCs of the pictures in the DPB are updated wrongly.

In general, layer up-switch points (IRAP pictures) need not co-exist inan access unit that has other pictures.

For example, a bitstream may contain coded texture video captured by aconventional color image sensor and coded depth video captured by aranging sensor. The sensors need not be tightly synchronized, i.e. theirpicture capture times may differ. Consequently, the texture and depthpictures may be placed in different access units. If one of the layers(usually the depth picture layer) is not transmitted for a period oftime, e.g. to adapt the transmitted bitrate, it may happen that when thenext IRAP picture in the depth layer is sent, prevTid0Pic of depth ismore than (MaxPicOrderCntLsb/2) apart. The POC decrementing wouldtherefore be performed wrongly.

In another example, similar to the one above, interlaced video is codedin a manner that e.g. the top field pictures form the base layer and theopposite field pictures form the enhancement layer. Hence, in any accessunit, there is only one picture present. If one of the layers (usuallythe enhancement layer) is not transmitted for a period of time, e.g. toadapt the transmitted bitrate, it may happen that when the next IRAPpicture in the enhancement layer is sent, prevTid0Pic of depth is morethan (MaxPicOrderCntLsb/2) apart. The POC decrementing would thereforebe performed wrongly.

In the layer-wise POC approach, layer down-switching causes the encoderand decoder to have different a prevTid0Pic at an enhancement layer IRAPpicture used to switch up. This is illustrated in FIG. 9. Both POC andlayer-wise POC derivation therefore become unpredictably wrong insubsequent enhancement layer pictures in decoding order.

Now in order to at least alleviate the above problems, an improvedmethod for determining a POC reference picture is provided herein.

According to a first aspect, which is illustrated in FIG. 10, picturesare encoded (1000) into a bitstream, the bitstream comprising at leasttwo scalability layers and pictures being associated with access units.An earlier picture in decoding order is selected (1000) as a basis forderiving picture order count (POC) related variables for a currentpicture based on a pre-defined algorithm, wherein the earlier picture issuch that it is required to be present in a bitstream.

In the next paragraphs various embodiments of selecting the earlierpicture are presented. Each embodiment may be applied independently, ortwo or more of the embodiments may be jointly used to specify conditionsall of which the earlier picture is required to fulfill, or two or moreembodiments may be jointly used to specified conditions at least one (orsome other pre-defined number) of which the earlier picture is requiredto fulfill, or conditions of two or more of the embodiments may becombined using one or more logical operations (such as AND, OR, XOR, andNOT) in certain order between the conditions and possibly the results ofthe previously computed logical operations.

According to an embodiment, the earlier picture is such that it isrequired for decoding of the current picture or the earlier picture isrequired to be present in the bitstream by bitstream conformanceconstraints.

According to an embodiment, the earlier picture is such that it isrequired for decoding of any picture in the target output layers of theoutput layer set in use or the earlier picture is required to be presentin the bitstream by bitstream conformance constraints. For example, theearlier picture may be such that it is selected among those picturesthat are not discardable pictures and/or NOLS pictures of layers thatare not among the target output layers.

According to an embodiment, the earlier picture is selected from anydirect or indirect reference layer of the layer of the current picture.Thus, the earlier picture need not necessarily be from the same layer asthe current picture.

According to an embodiment, the earlier picture is selected to be theprevious picture, in decoding order, from any direct or indirectreference layer of the layer of the current picture.

According to an embodiment, the earlier picture is selected to be theprevious picture, in decoding order, required to be present in thebitstream, such as a base-layer TRAP picture, even if it is not in adirect or indirect reference layer of the layer of the current picture.

According to an embodiment, the earlier picture is selected to be theprevious picture that cannot be extracted in a sub-bitstream extractionprocess creating a temporal subset of the bitstream. For example, theearlier picture may be selected to be a picture having TemporalId equalto 0 and may be required to be other than a RASL, RADL, or sub-layernon-reference picture.

According to an embodiment, the earlier picture is selected to be theprevious picture that can be decoded regardless of which picture thedecoding process is started from (as long as the decoding process isstarted before the earlier picture). For example, the earlier picturemay be selected to be a picture that is not a CL-RAS picture.

According to an embodiment, the earlier picture is selected to be theprevious picture that can be decoded regardless of any layerdown-switching that may have taken place before the current picture. Forexample, if the current picture enables layer up-switching, it may berequired that the earlier picture is not from the same layer as thecurrent picture.

A first example of combining embodiments of selecting the earlierpicture is as follows. The earlier picture is selected to be theprevious picture in decoding order for which all of the followingconditions are true:

-   -   The earlier picture has TemporalId equal to 0.    -   The earlier picture is not a RASL picture, a RADL picture, or a        sub-layer non-reference picture.    -   The earlier picture is not a discardable picture or a NOLS        picture in a layer that is not among the target output layers.    -   The earlier picture is not a CL-RAS picture.    -   At least one of the following conditions is true:        -   The current picture is not an IRAP picture and the earlier            has nuh_layer_id equal to nuh_layer_id of the current            picture.        -   The current picture is an IRAP picture in an independent            layer.        -   The earlier picture has nuh_layer_id among the direct or            indirect reference layers of the layer of the current            picture.

A second example of combining embodiments of selecting the earlierpicture is as follows. The earlier picture is selected to be theprevious picture in decoding order which either is an IRAP picture inthe base layer or an independent layer that is a direct or indirectreference layer of the current picture, or a picture that fulfills theconditions of the earlier picture in the first example above.

According to an embodiment, the POC coding related to the POC referencepicture is carried out such that at least one POC related syntax elementis encoded into the bitstream on the basis of the POC related variablesof said earlier picture defined as a POC reference picture.

According to an embodiment, the POC reference picture is used inDeltaPocVal derivation of the POC reset approach for decrementing thePOC values of the picture in a decoded picture buffer.

According to an embodiment, a layer tree is defined as an independentlayer and at least one dependent layer, wherein a mapping of layers tothe layer tree is obtained from the dependency information betweenlayers encoded into the bitstream. Layer-tree POC values of the samelayer tree are related to each other but the layer-tree POC values ofdifferent layer trees need not be related to each other. POC and/orlayer-tree POC related variables are derived on the basis of the POCreference picture.

According to an embodiment, the POC reference pictures may be selectedsuch that characteristics, including layer identifier(s), temporalsub-layer identifier(s), and/or picture types, of pictures on which thecurrent picture may depend on in prediction are determined, and theprevious picture, in decoding order, fulfilling the characteristics isselected as the POC reference picture.

The decoding operations are opposite to the encoding operationsdescribed above. Thus, according to another aspect, which is illustratedin FIG. 11, there is provided a decoding method, where pictures aredecoded (1100) from a bitstream, the bitstream comprising at least twoscalability layers and pictures being associated with access units. Anearlier picture in decoding order is selected as a basis for derivingpicture order count (POC) related variables for a current picture basedon a pre-defined algorithm, wherein the earlier picture is such that itis required to be present in a bitstream.

An example of implementing the invention with the POC reset approach ispresented below.

If FirstPicInLayerDecodedFlag[nuh_layer_id] is equal to 1, poc_reset_idcis greater than 0, and the access unit containing the current picture isthe first access unit in decoding order in a POC resetting period, thefollowing applies:

-   -   The variables pocMsbDelta, pocLsbDelta and DeltaPocVal are        derived as follows:

prevPicOrderCntLsb = PrevPicOrderCnt[ nuh_layer_id ] & (MaxPicOrderCntLsb − 1 ) prevPicOrderCntMsb = PrevPicOrderCnt[nuh_layer_id ] − prevPicOrderCntLsb if( poc_reset_idc == 3 ∥ (poc_reset_idc == 2 && nuh_layer_id == 0 ) ) pocLsbVal = poc_lsb_val elsepocLsbVal = slice_pic_order_cnt_lsb pocMsbDelta = getCurrMsb( pocLsbVal,prevPicOrderCntLsb, prevPicOrderCntMsb, MaxPicOrderCntLsb ) if(poc_reset_idc == 2 ∥ ( poc_reset_idc == 3 && full_poc_reset_flag ) )pocLsbDelta = pocLsbVal else pocLsbDelta = 0 DeltaPocVal = pocMsbDelta +pocLsbDelta

-   -   The PicOrderCntVal of the current picture is derived as follows:

if( poc_reset_idc == 1 ) PicOrderCntVal = slice_pic_order_cnt_lsb elseif( poc_reset_idc == 2 ) PicOrderCntVal = 0 else { // poc_reset_idc == 3PicOrderCntMsb = getCurrMsb( slice_pic_order_cnt_lsb,full_poc_reset_flag ? 0 : poc_lsb_val, 0, MaxPicOrderCntLsb )PicOrderCntVal = PicOrderCntMsb + slice_pic_order_cnt_lsb }

Now in contrast to the prior known approach, the value ofPrevPicOrderCnt[lId] for lId equal to nuh_layer_id of the currentpicture for each layer with nuh_layer_id value lId for which the currentlayer is a direct or indirect reference layer is derived as follows:

-   -   If the current picture is not a RASL picture, a RADL picture, or        a sub-layer non-reference picture, and the current picture has        TemporalId equal to 0, PrevPicOrderCnt[lId] is set equal to        PicOrderCntVal.    -   Otherwise when poc_reset_idc is equal to 3, PrevPicOrderCnt[lId]        is set equal to full_poc_reset_flag ? 0:poc_lsb_val.

Otherwise, the following applies:

-   -   The value of DeltaPocVal is set equal to 0.    -   The PicOrderCntVal of the current picture is derived as follows:

if( !FirstPicInLayerDecodedFlag[ nuh_layer_id ] ) { if( poc_reset_idc ==1 ) PicOrderCntVal = slice_pic_order_cnt_lsb else if( poc_reset_idc == 2) PicOrderCntVal = 0 else if( poc_reset_idc == 3 ) { PicOrderCntMsb =getCurrMsb(slice_pic_order_cnt_lsb, full_poc_reset_flag ? 0 :poc_lsb_val, 0, MaxPicOrderCntLsb ) PicOrderCntVal = PicOrderCntMsb +slice_pic_order_cnt_lsb } else // the current picture is an IRAP picturewith NoRaslOutputFlag equal to 1 PicOrderCntVal =slice_pic_order_cnt_lsb } else { // the POC derivation as in HEVCversion 1 if( the current picture is an IRAP picture withNoRaslOutputFlag equal to 1 ) PicOrderCntMsb = 0 else {prevPicOrderCntLsb = PrevPicOrderCnt[ nuh_layer_id ] & (MaxPicOrderCntLsb − 1 ) prevPicOrderCntMsb = PrevPicOrderCnt[nuh_layer_id ] − prevPicOrderCntLsb PicOrderCntMsb = getCurrMsb(slice_pic_order_cnt_lsb, prevPicOrderCntLsb, prevPicOrderCntMsb,MaxPicOrderCntLsb ) } PicOrderCntVal = PicOrderCntMsb +slice_pic_order_cnt_lsb }

-   -   The value of PrevPicOrderCnt[lId] for lId equal to nuh_layer_id        of the current picture for each layer with nuh_layer_id value        lId for which the current layer is a direct or indirect        reference layer is now derived as follows:        -   If the current picture is not a RASL picture, a RADL            picture, or a sub-layer non-reference picture, and the            current picture has TemporalId equal to 0,            PrevPicOrderCnt[lId] is set equal to PicOrderCntVal.        -   Otherwise when FirstPicInLayerDecodedFlag[nuh_layer_id] is            equal to 0 and poc_reset_idc is equal to 3,            PrevPicOrderCnt[lId] is set equal to full_poc_reset_flag ?            0:poc_lsb_val.

If the current picture is the first picture in an access unit (i.e. thepicture with the lowest value of nuh_layer_id among all pictures in theaccess unit) and DeltaPocVal is greater than 0, the PicOrderCntValvalues of all decoded pictures of all layers in the DPB are decrementedby the value of DeltaPocVal and PrevPicOrderCnt[lId] is decremented bythe value of DeltaPocVal for each value of lId in the range of 0 toMaxLayersMinus1, inclusive.

The layer-tree POC approach may be implemented as follows:

1. A layer tree is defined to consist of one independent layer (withoutany reference layers) and all layers that directly or indirectly use theindependent layer as reference.

2. Slice header extensions optionally contain signalling for alayer-tree POC value, which is intended to be used in the IRAP picturesof the independent layers and may be used, for additional errorresilience, in other pictures too.

3. A layer-tree POC value, LayerTreePicOrderCntVal is derived for eachpicture based on the mentioned slice header signalling, if present, andbased on PicOrderCntVal differences, otherwise.

4. Both the POC value and the layer-tree POC value are derived relativeto the POC value and the layer-tree POC, respectively, of the POCreference picture. The layer-tree POC values of all pictures in the samelayer tree in the same access unit are equal.

5. In some embodiments (with POC reset signaling), enhancement-layerslice headers optionally contain POC resetting information. Thissignalling may be required to be used for a picture that is in adifferent layer tree than the layer tree containing the base layer andwhich is the first picture, in decoding order, in a layer tree after abase-layer IDR or BLA picture. The signalling may also be used in otherpictures for additional error resilience.

6. The function DiffPicOrderCnt( ) is specified to useLayerTreePicOrderCntVal values.

Some examples of syntax, semantics, and decoding operation for thelayer-tree POC approach are provided in the following. The examplesshare some common definitions, which are presented first below, followedby specific examples of syntax, semantics and related decodingoperation.

The following definitions are used in the layer-tree POC examples:

-   -   A direct reference layer may refer to a layer that may be used        for inter-layer prediction of the current layer (i.e. the layer        of the current picture or another layer to which the “direct”        reference relation is indicated).    -   An indirect reference layer may refer to a layer that is not a        direct reference layer of a first layer but is a direct        reference layer of a layer that is a direct reference layer or        indirect reference layer of a direct reference layer of the        first layer.    -   An independent layer may refer to a layer that does not have        direct reference layers.    -   A layer tree may refer to a set of layers consisting of an        independent layer and all layers for which the independent layer        is a direct reference layer or an indirect reference layer.    -   A layer-tree picture order count may refer to a variable that is        associated with each picture, uniquely identifies the access        unit containing the picture within the layer tree containing the        picture within a sequence of access units in decoding order from        a picture with NoClrasOutputFlag equal to 1 in the independent        layer of the layer tree, inclusive, to the next picture with        NoClrasOutputFlag equal to 1 in the independent layer of the        layer tree, exclusive, or the end of the bitstream, whichever is        earlier in decoding order.

In the layer-tree POC examples, the functionmore_data_in_slice_segment_header_extension( ) may be specified asfollows: If (the current position in the slice_segment_header( ) syntaxstructure)−(the position immediately followingslice_segment_header_extension_length) is less than(slice_segment_header_extension_length*8), the return value ofmore_data_in_slice_segment_header_extension( ) is equal to TRUE.Otherwise, the return value ofmore_data_in_slice_segment_header_extension( ) is equal to FALSE. Here,the position refers to the bit position from which the data is read andparsed from the bitstream.

An example of the layer-tree POC approach with POC reset signaling isprovided in the following. The slice segment header syntax table may beappended for example as follows (while other possibilities ofconditioning the presence of poc_reset_flag, poc_msb_cycle orlayer_tree_poc_lsb also exist). In this example,layer_tree_poc_info_present_flag is conveyed in the picture parameterset syntax structure.

Des- crip- slice_segment_header( ) { tor   ...   if( nuh_layer_id > 0 ){    poc_reset_flag u(1)    if( poc_reset_flag)     poc_msb_cycle se(v)  }   ...  if( slice_segment_header_extension_present_flag ) {  slice_segment_header_extension_length ue(v)   if(layer_tree_poc_info_present_flag )    layer_tree_poc_lsb u(v)   while(more_data_in_slice_segment_header_extension( ) )   slice_segment_header_extension_data_bit u(1)  }  byte_alignment( ) }

Continuing the example, the derivation of POC and layer-tree POC valuesin this approach (with POC reset signaling using poc_msb_cycle) may bespecified as follows:

Let prevTid0Pic be the previous picture in decoding order for which allof the following conditions are true:

-   -   prevTid0Pic has TemporalId equal to 0.    -   prevTid0Pic is not a RASL picture, a RADL picture, or a        sub-layer non-reference picture.    -   At least one of the following conditions is true:        -   The current picture is not an IRAP picture and prevTid0Pic            has nuh_layer_id equal to nuh_layer_id of the current            picture.        -   The current picture is an IRAP picture in an independent            layer.        -   prevTid0Pic has nuh_layer_id equal to            RecursiveRefLayerId[nuh_layer_id][i] for any value of i in            the range of 0 to NumRefLayers[nuh_layer_id]−1, inclusive.            (Here RecursiveRefLayerId[nuhLayerId] is a list of            nuh_layer_id values of direct and indirect reference layers            of the layer with nuh_layer_id equal to nuhLayerId, and            NumRefLayers[nuhLayerId] is the number of direct and            indirect reference layers of the layer with nuh_layer_id            equal to nuhLayerId.)

The variable PicOrderCntMsb is derived as follows:

-   -   If the current picture is an IRAP picture with NoRaslOutputFlag        equal to 1 and with nuh_layer_id equal to 0, PicOrderCntMsb is        set equal to 0.    -   Otherwise, if poc_reset_flag is equal to 1 (or a base-layer POC        reset is detected by other means), PicOrderCntMsb is set equal        to poc_msb_cycle*MaxPicOrderCntLsb.    -   Otherwise, the following applies        -   Let prevPicOrderCnt be equal to PicOrderCntVal of            prevTid0Pic.        -   The variable prevPicOrderCntLsb is set equal to            slice_pic_order_cnt_lsb of prevTid0Pic.        -   The variable prevPicOrderCntMsb is set equal to            PicOrderCntMsb of prevTid0Pic.        -   PicOrderCntMsb is derived as follows:

if( ( slice_pic_order_cnt_lsb < prevPicOrderCntLsb ) && ( (prevPicOrderCntLsb − slice_pic_order_cnt_lsb ) >= ( MaxPicOrderCntLsb /2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb + MaxPicOrderCntLsb else if((slice_pic_order_cnt_lsb > prevPicOrderCntLsb ) && ((slice_pic_order_cnt_lsb − prevPicOrderCntLsb ) > ( MaxPicOrderCntLsb /2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb − MaxPicOrderCntLsb elsePicOrderCntMsb = prevPicOrderCntMsb

PicOrderCntVal is derived as follows:PicOrderCntVal=PicOrderCntMsb+slice_pic_order_cnt_lsb

The variable LayerTreePicOrderCntVal is derived as follows:

-   -   If the current picture is an IRAP picture with NoClrasOutputFlag        equal to 1 in an independent layer, LayerTreePicOrderCntVal is        set equal to layer_tree_poc_lsb.    -   Otherwise, the following applies:        -   Let prevLayerTreePicOrderCnt be equal to            LayerTreePicOrderCntVal of prevTid0Pic.        -   If layer_tree_poc_info_present_flag is equal to 1, the            following applies:        -   The variable prevLayerTreePicOrderCntLsb is set equal to            prevLayerTreePicOrderCnt & (MaxPicOrderCntLsb−1).        -   The variable prevLayerTreePicOrderCntMsb is set equal to            prevLayerTreePicOrderCnt−prevLayerTreePicOrderCntLsb.        -   The variable layerTreePicOrderCntMsb is derived as follows:

if( ( layer_tree_poc_lsb < prevLayerTreePicOrderCntLsb ) && ( (prevLayerTreePicOrderCntLsb − layer_tree_poc_lsb ) >= (MaxPicOrderCntLsb / 2 ) ) ) layerTreePicOrderCntMsb =prevLayerTreePicOrderCntMsb + MaxPicOrderCntLsb) else if( (layer_tree_poc_lsb > prevLayerTreePicOrderCntLsb ) && ( (layer_tree_poc_lsb − prevLayerTreePicOrderCntLsb ) > ( MaxPicOrderCntLsb/ 2 ) ) ) layerTreePicOrderCntMsb = prevLayerTreePicOrderCntMsb −MaxPicOrderCntLsb else layerTreePicOrderCntMsb =prevLayerTreePicOrderCntMsb

-   -   -   LayerTreePicOrderCntVal is set equal to            layerTreePicOrderCntMsb+layer_tree_poc_lsb.

    -   Otherwise, LayerTreePicOrderCntVal is set equal to        prevLayrTreePicOrderCnt+PicOrderCntVal−prevPicOrderCnt.

The function layerTreePicOrderCnt(picX) is specified as follows:layerTreePicOrderCnt(picX)=LayerTreePicOrderCntVal of the picture picX

The function DiffPicOrderCnt(picA, picB) is specified as follows:DiffPicOrderCnt(picA,picB)=layerTreePicOrderCnt(picA)−layerTreePicOrderCnt(picB)

Another example of the layer-tree POC approach with POC reset signalingis provided in the following. The slice segment header syntax table maybe appended for example as follows (while other possibilities ofconditioning the presence of poc_reset_flag, ref poc_lsb orlayer_tree_poc_lsb also exist). In this example,layer_tree_poc_info_present_flag is conveyed in the picture parameterset syntax structure.

Des- crip- slice_segment_header( ) { tor   ...   if( nuh_layer_id > 0 ){    poc_reset_flag u(1)    if( poc_reset_flag )     ref_poc_lsb se(v)  }   ...  if( slice_segment_header_extension_present_flag ) {  slice_segment_header_extension_length ue(v)   if(layer_tree_poc_info_present_flag )    layer_tree_poc_lsb u(v)   while(more_data_in_slice_segment_header_extension( ) )   slice_segment_header_extension_data_bit u(1)  }  byte_alignment( ) }

In this example, prevTid0Pic may be derived for example identically tothe previous example.

The variable PicOrderCntMsb is derived as follows:

-   -   If the current picture is an IRAP picture with NoRaslOutputFlag        equal to 1 and with nuh_layer_id equal to 0, PicOrderCntMsb is        set equal to 0.    -   Otherwise, the following applies:        -   If poc_reset_flag is equal to 1, the following applies:            -   The variable prevPicOrderCntLsb is set equal to ref                poc_lsb.            -   The variable prevPicOrderCntMsb is set equal to 0.        -   Otherwise (poc_reset_flag is equal to 0), the following            applies:            -   Let prevPicOrderCnt be equal to PicOrderCntVal of                prevTid0Pic.            -   The variable prevPicOrderCntLsb is set equal to                slice_pic_order_cnt_lsb of prevTid0Pic.            -   The variable prevPicOrderCntMsb is set equal to                PicOrderCntMsb of prevTid0Pic.        -   PicOrderCntMsb is derived as follows:

if( ( slice_pic_order_cnt_lsb < prevPicOrderCntLsb ) && ( (prevPicOrderCntLsb − slice_pic_order_cnt_lsb ) >= ( MaxPicOrderCntLsb /2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb + MaxPicOrderCntLsb else if((slice_pic_order_cnt_lsb > prevPicOrderCntLsb ) && ((slice_pic_order_cnt_lsb − prevPicOrderCntLsb ) > ( MaxPicOrderCntLsb /2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb − MaxPicOrderCntLsb elsePicOrderCntMsb = prevPicOrderCntMsb

PicOrderCntVal is derived as follows:PicOrderCntVal=PicOrderCntMsb+slice_pic_order_cnt_lsb

The variable LayerTreePicOrderCntVal as well as the functionDiffPicOrderCnt may be derived identically to the layer-tree POCapproach with POC reset signalling (with poc_msb_cycle).

An example of the layer-tree POC approach without POC reset signaling isprovided in the following. The slice segment header syntax table may beappended for example as follows (while other possibilities ofconditioning the presence of layer_tree_poc_lsb also exist). In thisexample, layer_tree_poc_info_present_flag is conveyed in the pictureparameter set syntax structure.

Des- crip- slice_segment_header( ) { tor   ...  if(slice_segment_header_extension_present_flag ) {  slice_segment_header_extension_length ue(v)   if(layer_tree_poc_info_present_flag )    layer_tree_poc_lsb u(v)   while(more_data_in_slice_segment_header_extension( ) )   slice_segment_header_extension_data_bit u(1)  }  byte_alignment( ) }

In the layer-tree POC approach without POC reset signaling, the encoderand/or the decoder may keep track of when a base-layer picture resetsthe POC value. For example, base-layer IDR and BLA pictures reset thePOC value (to 0 or to the signaled slice_pic_order_cnt_lsb,respectively). More generally, a base-layer picture withNoRaslOutputFlag equal to 1 can be considered to reset the POC value. Asresponse to a base-layer picture resetting the POC value, the encoderand/or the decoder may be keep track if the POC reset has been takeninto account for the first picture in each enhancement layer followingthe base-layer picture resetting the POC value. This may be done forexample by maintaining a variable BLPOCResetFlag[nuh_layer_id] fordifferent nuh_layer_id values. The variable may be set as follows asresponse to a base-layer picture resetting the POC value: When thecurrent picture is an TRAP picture with nuh_layer_id equal to 0 andNoRaslOutputFlag equal to 1, the variable BLPOCResetFlag[i] is set equalto 1 for all values of i in the range of 1 to MaxLayersMinus1,inclusive.

In some embodiments which may be applied together with or independentlyof other embodiments, a base-layer POC reset may be additionally oralternatively identified with one or more of the following means, whichmay be used particularly for resilience against full base-layer picturelosses:

-   -   Pictures may be assigned poc_reset_period_id (similarly to the        POC reset approach), which may be signaled for selected pictures        as determined by the encoder. As in the layer-tree POC approach        POC is reset only by base-layer picture, a change in the        poc_reset_period_id can be used to conclude a base-layer POC        reset.    -   Pictures may be assigned bl_cvs_id which may be signaled for        selected pictures as determined by the encoder. The bl_cvs_id        shall differ in subsequent (in decoding order) IDR and BLA        pictures of the base layer. A change in the bl_cvs_id can be        used to conclude a base-layer POC reset.    -   Pictures may be assigned layer_tree_cvs_id, which may be        signaled for selected pictures as determined by the encoder.        layer_tree_cvs_id shall differ in in subsequent (in decoding        order) IDR and BLA pictures of the same independent layer. A        change in the layer_tree_cvs_id for the base layer can be used        to conclude a base-layer POC reset.    -   Temporal sub-layer zero index SEI message of H.265/HEVC or a        similar SEI message or other syntax structure may be used to        associate pictures to IRAP access units. If a temporal sub-layer        zero index SEI message indicates an IRAP access unit for which        the base-layer picture has not been received or decoded (at all        or correctly), a loss of a base-layer IRAP picture can be        concluded.

Continuing the example of the layer-tree POC approach without POC resetsignaling, the decoding process may operate as follows. The prevTid0Picmay be derived identically to the other examples of the layer-tree POCapproach. The variable PicOrderCntMsb is derived as follows:

-   -   If the current picture is an IRAP picture with NoRaslOutputFlag        equal to 1 and with nuh_layer_id equal to 0, PicOrderCntMsb is        set equal to 0.    -   Otherwise, the following applies:        -   If BLPOCResetFlag[nuh_layer_id] is equal to 0 or prevTid0Pic            (if any) is subsequent, in decoding order, to the previous,            in decoding order, IRAP picture with nuh_layer equal to 0            and NoRaslOutputFlag equal to 1, the variable            prevPicOrderCnt is set equal to PicOrderCntVal of            prevTid0Pic        -   Otherwise (BLPOCResetFlag[nuh_layer_id] is equal to 1), the            variable prevPicOrderCnt is set equal to to PicOrderCntVal            of the previous, in decoding order, IRAP picture with            nuh_layer equal to 0 and NoRaslOutputFlag equal to 1.        -   The variable prevPicOrderCntLsb is set equal to            prevPicOrderCnt & (MaxPicOrderCntLsb−1).        -   The variable prevPicOrderCntMsb is set equal to            prevPicOrderCnt−prevPicOrderCntLsb.        -   PicOrderCntMsb is derived as follows:

if( ( slice_pic_order_cnt_lsb < prevPicOrderCntLsb ) && ( (prevPicOrderCntLsb − slice_pic_order_cnt_lsb ) >= ( MaxPicOrderCntLsb /2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb + MaxPicOrderCntLsb else if((slice_pic_order_cnt_lsb > prevPicOrderCntLsb ) && ((slice_pic_order_cnt_lsb − prevPicOrderCntLsb ) > ( MaxPicOrderCntLsb /2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb − MaxPicOrderCntLsb elsePicOrderCntMsb = prevPicOrderCntMsb

PicOrderCntVal is derived as follows:PicOrderCntVal=PicOrderCntMsb+slice_pic_order_cnt_lsb

Subsequently, BLPOCResetFlag[nuh_layer_id] is set equal to 0.

The variable LayerTreePicOrderCntVal as well as the functionDiffPicOrderCnt may be derived identically to the layer-tree POCapproach with POC reset signalling (with poc_msb_cycle).

In the layer-tree POC approach, different functionalities related to POCmay be implemented as follows:

-   -   The decoding processes (e.g. motion vector scaling in TMVP) may        be use the POC value or the layer-tree POC value in the        base-layer decoding. For decoding of enhancement layers, the        difference of layer-tree POCs may be used. For decoding of the        enhancement layers, the function DiffPicOrderCnt(picA, picB) is        specified to use layer-tree POCs.    -   An access unit (AU) boundary may be detected using the        pic_order_cnt_lsb or the derived POC value. Consecutive pictures        having the same POC or the same pic_order_cnt_lsb are contained        in the same access unit. An AU boundary can be concluded when i)        there is a picture with first_slice_segment_in_pic_flag equal to        1 and nuh_layer_id equal to 0, or ii) slice_pic_order_cnt_lsb        differs in adjacent pictures.    -   Reference picture sets in the base layer may be signaled using        POC values or the layer-tree POC values. Similarly, reference        picture sets in the enhancement layers may be signaled using POC        values or layer-tree POC values, although the signaling may be        more efficient when layer-tree POC values are used (as no        unintentional discontinues of the used values then exist).    -   Picture output order determination in the bumping process of the        HRD basically follows the base-layer output order determination.        Access units of a CVS are contiguous in output order. The output        order determination is first based on the CVS order, i.e. all        access units of an earlier CVS are output before any access unit        of a later CVS. Within a CVS, the access unit output order is        determined by PicOrderCntVal. All the pictures of an access unit        (with PicOutputFlag equal to 1) are output contiguously.    -   SEI messages applying to base-layer pictures or entire access        units may use PicOrderCntVal in their semantics. SEI messages        with nuh_layer_id greater than 0 or included in a scalable        nesting SEI message may use LayerTreePicOrderCntVal in their        semantics when the SEI message applies to enhancement-layer        pictures.    -   For feedback based error control or recovery, PicOrderCntVal may        be used for identifying base-layer pictures and        LayerTreePicOrderCntVal may be used for identifying        enhancement-layer pictures in the feedback messages.

The Reference Picture Selection Indication (RPSI) payload of RTP/AVPF(IETF RFC 4585) may be specified for MV-HEVC/SHVC streams as follows:The field “Native RPSI bit string defined per codec” is a base16 [IETFRFC 4648] representation of the 8 bits consisting of 2 most significantbits equal to 0 and 6 bits of nuh_layer_id, as defined in H.265/HEVC,followed by the 32 bits representing the value of the PicOrderCntVal (innetwork byte order) when nuh_layer_id is equal to 0 or the 32 bitsrepresenting the LayerTreePicOrderCntVal (in network byte order), forthe picture that is requested to be used for reference when encoding thenext picture.

In the above, the functionalities related to POC may alternatively referto an independent layer when term base layer is used and to a dependentlayer (a layer which is not an independent layer) when term enhancementlayer is used.

In the following, several examples of the layer-tree POC approach arepresented.

In the example of FIG. 12, there is access unit with an IDR picture inthe base layer, which resets the POC value to 0. The layer-tree POCvalues are anyhow continued by including the layer-tree POC relatedsyntax elements (e.g. layer_tree_poc_lsb) with the base-layer IDRpicture. In this example layer_tree_poc_lsb is equal to 10, and sincethe derived layer-tree POC for the POC reference picture is 1032 andconsequently the layer-tree POC LSB is equal to 8 (whenMaxPicOrderCntLsb is equal to 16), the layer-tree POC value isincremented by 2 for the base-layer IDR picture.

In the example of FIG. 13, additional resilience against the loss of abase-layer IDR picture is provided so that the loss of a base-layer IDRpicture does not cause derivation of wrong POC values or layer-tree POCvalues. POC reset related information is included with theenhancement-layer picture in the access unit containing base-layer IDRpicture. For example, poc_reset_flag is set to 1 and poc_msb_cycle isset to 0, while if the alternative of signaling ref_poc_lsb was used,the value of ref poc_lsb would be set to 0 in the presented enhancementlayer picture with poc_reset_flag equal to 1. This informationfacilitates correct POC value derivation even if the base-layer IDRpicture was lost. Moreover, layer-tree POC information (e.g.layer_tree_poc_lsb) is included with the same enhancement-layer picture.This information facilitates correct layer-tree POC value derivationeven if the base-layer IDR picture was lost.

In the example of FIG. 14, the derivation of a POC value and alayer-tree POC value for an enhancement-layer picture that is present inan access unit that does not include a base-layer picture and follows(in decoding order) an access unit that resets the POC value. As in theprevious example figures, the base-layer IDR picture includeslayer_tree_poc_lsb (or alike) that enables the derivation of thelayer-tree POC value for the base-layer IDR picture. The value of Inthis example, the POC reference picture for the first enhancement layerpicture following a POC reset point is set to be the preceding (indecoding order) base-layer picture. The base-layer picture has both POCvalue and layer-tree POC value and serves as reference for both POCvalue and layer-tree POC value derivation for the enhancement-layerpicture.

The example of FIG. 15 illustrates a reset of layer-tree POC values. Inthis example, the layer-tree POC values are reset by including across_layer_bla_flag equal to 1 in a base-layer IDR picture. Thebase-layer IDR picture (regardless of the value of cross_layer_bla_flag)also resets the POC value.

In the example of FIG. 16, there are two independent layers. Hence, thelayer-tree POC value derivation of each layer is independent of theother layer (while in the example the layer-tree POC values of bothlayers happen to indicate their joint output order). The enhancementlayer picture that follows, in decoding order, the base-layer IDRpicture has poc_reset_flag equal to 1. The POC reference picture(prevTid0Pic) for the enhancement-layer picture with layer-tree POCequal to 1035 is the enhancement-layer picture with layer-tree POC equalto 1033.

The example of FIG. 17 presents an IDR picture in the enhancement layer,which may be used for layer up-switching. The POC reference picture forthis enhancement-layer IDR picture is the base-layer picture in the sameaccess unit. FIG. 18 illustrates the same bitstream as in FIG. 17, butwhere a layer down-switch has happened prior to the enhancement-layerIDR picture and a layer up-switch is taking place at theenhancement-layer IDR picture. As the POC reference picture is thebase-layer picture, both the POC value and the layer-tree POC value ofthe enhancement-layer IDR picture can be correctly decoded without theneed of receiving or decoding earlier pictures (in decoding order) ofthe enhancement layer. FIG. 19 illustrates a similar bitstream as inFIG. 17, where the enhancement layer was not present or has been removedpreceding (in decoding order) the enhancement-layer IDR picture.Similarly, as the POC reference picture is the base layer picture in theaccess unit that also contains the enhancement-layer IDR picture, boththe POC value and the layer-tree POC value of the enhancement-layer IDRpicture can be correctly decoded.

FIG. 20 illustrates an ARC bitstream that includes a switch to theenhancement-layer resolution and then back to the base-layer resolution.Similarly to the example of FIG. 19, the POC reference picture of theenhancement-layer IDR picture is the base layer picture in the accessunit that also contains the enhancement-layer IDR picture, both the POCvalue and the layer-tree POC value of the enhancement-layer IDR picturecan be correctly decoded. In this example, the base-layer picture wherethe switching back to the base-layer resolution happens is sufficientlyclose to the preceding base-layer picture (in decoding order) in termsof the POC values, such that the difference of PicOrderCntVal values isless than MaxPicOrderCntLsb/2, and consequently the update ofPicOrderCntMsb is concluded correctly.

FIG. 21 illustrates an ARC bitstream that includes a switch to theenhancement-layer resolution and then back to the base-layer resolution.In this case example, the base-layer picture where the switching back tothe base-layer resolution is coded as an IDR picture withcross_layer_bla_flag equal to 1 so that no POC reference picture is usedin the POC or layer-tree POC derivation. A BLA picture could besimilarly used instead of an IDR picture with cross_layer_bla_flag equalto 1.

In the presented example figures above, where poc_reset_flag equal to 1is referred, other options for signaling or determining a base-layer POCreset may be used alternatively or additionally. For example,BLPOCResetFlag[nuh_layer_id] equal to 1 could be used instead of or inaddition to poc_reset_flag equal to 1.

In an embodiment which may be applied independently of or together withother embodiments a decoder or a decoder side entity creates feedbackmessages for POC values derived using the POC reset approach. Picturesare identified in the feedback messages with a combination of i) apoc_reset_period_id, bl_cvs_id, irap_pic_id (for the base layer),layer_tree_cvs_id (for the base layer) or similar and ii) the POC value.The poc_reset_period_id or similar enables the recipient of the feedbackmessages to synchronize to the state of the decoder (or decoder sideentity) when the message was created. For example, the encoder maymaintain a set of POC values for pictures (that it is buffering, e.g. inits DPB), one POC value for each poc_reset_period_id that has takenplace at or after the picture in decoding order. The encoder can thencompare the combination of poc_reset_period_id and POC value with thestored combinations of poc_reset_period_id and POC value for eachpicture and conclude which picture the decoder (or the decoder sideentity) refers to.

For example, the Reference Picture Selection Indication (RPSI) payloadof RTP/AVPF (IETF RFC 4585) may be specified for MV-HEVC/SHVC streams asfollows: The field “Native RPSI bit string defined per codec” is abase16 [IETF RFC 4648] representation of poc_reset_period_id (8 bitswhere the value appears in the LSBs) followed by the value of thePicOrderCntVal (in network byte order), for the picture that isrequested to be used for reference when encoding the next picture.

The encoder or another entity, such as an HRD verifier, may indicatebuffering parameters for one or both of the following types ofbitstreams: bitstreams where CL-RAS pictures of IRAP pictures for whichNoClrasOutputFlag is equal to 1 are present and bitstreams where CL-RASpicture of IRAP pictures for which NoClrasOutputFlag is equal to 1 arenot present. For example, CPB buffer size(s) and bitrate(s) may beindicated separately e.g. in VUI for either or both mentioned types ofbitstreams. Additionally or alternatively, the encoder or another entitymay indicate initial CPB and/or DPB buffering delay and/or otherbuffering and/or timing parameters for either or both mentioned types ofbitstreams. The encoder or another entity may, for example, include abuffering period SEI message into a scalable nesting SEI message, whichmay indicate the sub-bitstream, the layer set or the output layer set towhich the contained buffering period SEI message applies. The bufferingperiod SEI message of HEVC supports indicating two sets of parameters,one for the case where the leading pictures associated with the IRAPpicture (for which the buffering period SEI message is also associatedwith) are present and another for the case where the leading picturesare not present. In the case when a buffering period SEI message iscontained within a scalable nesting SEI message, the latter(alternative) set of parameters may be considered to concern a bitstreamwhere CL-RAS pictures associated with the IRAP picture (for which thebuffering period SEI message is also associated with) are not present.Generally, the latter set of buffering parameters may concern abitstream where CL-RAS pictures associated with an IRAP picture forwhich NoClrasOutputFlag is equal to 1 are not present. It is to beunderstood that while specific terms and variable names are used in thedescription of this paragraph, it can be similarly realized with otherterminology and need not use the same or similar variables as long asthe decoder operation is similar.

The encoder or another entity, such as an HRD verifier, may indicatebuffering parameters for output layer sets within which NOLS picturesare removed on layers that are not output. In other words, bufferingparameters may be indicated for a sub-bitstream containing the layersincluded in an output layer set where NOLS pictures of the layers notamong the target output layers of the output layer set are removed. Forexample, CPB buffer size(s) and bitrate(s) may be indicated e.g. in VUIfor such sub-bitstreams. Additionally or alternatively, the encoder oranother entity may indicate initial CPB and/or DPB buffering delayand/or other buffering and/or timing parameters for such sub-bitstreams.The encoder or another entity may, for example, include a bufferingperiod SEI message into a nesting SEI message, which may indicate theoutput layer set to which the contained buffering period SEI messageapplies. The buffering period SEI message of HEVC supports indicatingtwo sets of parameters, one for the case where the leading picturesassociated with the IRAP picture (for which the buffering period SEImessage is also associated with) are present and another for the casewhere the leading pictures are not present. In the case when a bufferingperiod SEI message is contained within a nesting SEI message applying toan output layer set, the latter (alternative) set of parameters may beconsidered to concern a bitstream where NOLS pictures of non-outputlayers and/or CL-RAS pictures associated with the IRAP picture (forwhich the buffering period SEI message is also associated with) are notpresent. Generally, the latter set of buffering parameters may concern abitstream where CL-RAS pictures associated with an IRAP picture forwhich NoClrasOutputFlag is equal to 1 are not present. It is to beunderstood that while specific terms and variable names are used in thedescription of this embodiment, it can be similarly realized with otherterminology and need not use the same or similar variables as long asthe decoder operation is similar.

A sub-bitstream extraction process where inputs are a bitstream, anoutput layer set (or a list of output layers), and possibly a maximumTemporalId value (highestTid) may be used. An output layer set may beused to infer a target layer set that is required to be decoded, or thetarget layer set may be provided as input. An output of the process is asub-bitstream. The sub-bitstream extraction process includes in thesub-bitstream all VCL NAL units within the output layer set havingTemporalId smaller than or equal to highestTid and the non-VCL NAL unitsassociated with the included VCL NAL units. Furthermore, thesub-bitstream extraction process included those VCL NAL units from thenon-output layers (but included in the target layer set) that are notNOLS pictures and not discardable pictures. Furthermore, thesub-bitstream extraction process may modify the syntax structurecontaining information on output layer sets, such as VPS, to excludethose output layer sets where the layers from which NOLS pictures anddiscardable pictures are on output layers. Furthermore, thesub-bitstream extraction process may modify the signaling of HRDparameters such that the HRD parameters that apply for output layer setswithout the presence of NOLS pictures (and discardable pictures) of thenon-output layers are in force. Furthermore, the sub-bitstreamextraction process may exclude those HRD parameters from the bitstreamthat assume the presence of NOLS pictures (and discardable pictures) ofthe non-output layers.

A sub-bitstream extraction process may exclude CL-RAS picturesassociated with TRAP pictures for which NoClrasOutputFlag is equal to 1.Furthermore, the sub-bitstream extraction process may exclude those HRDparameters from the bitstream that assume the presence of CL-RASpictures that have been removed from the output bitstream.

In the above, some embodiments have been described in relation toencoding indications, syntax elements, and/or syntax structures into abitstream or into a coded video sequence and/or decoding indications,syntax elements, and/or syntax structures from a bitstream or from acoded video sequence. It needs to be understood, however, thatembodiments could be realized when encoding indications, syntaxelements, and/or syntax structures into a syntax structure or a dataunit that is external from a bitstream or a coded video sequencecomprising video coding layer data, such as coded slices, and/ordecoding indications, syntax elements, and/or syntax structures from asyntax structure or a data unit that is external from a bitstream or acoded video sequence comprising video coding layer data, such as codedslices.

In the above, the example embodiments have been described with the helpof syntax of the bitstream. It needs to be understood, however, that thecorresponding structure and/or computer program may reside at theencoder for generating the bitstream and/or at the decoder for decodingthe bitstream. Likewise, where the example embodiments have beendescribed with reference to an encoder, it needs to be understood thatthe resulting bitstream and the decoder have corresponding elements inthem. Likewise, where the example embodiments have been described withreference to a decoder, it needs to be understood that the encoder hasstructure and/or computer program for generating the bitstream to bedecoded by the decoder.

Where example embodiments have been described with reference to asplicer, it needs to be understood that a splicer could likewise be anencoder, a middle-box, or any other entity that creates or modifies acoded video bitstream.

In the above, some embodiments have been described with reference to anenhancement layer and a reference layer, where the reference layer maybe for example a base layer. In the above, some enhancements have beendescribed with reference to an enhancement layer and a base layer, wherethe base layer may be considered to be any reference layer of theenhancement layer.

It needs to be understood that embodiments may be applicable to anytypes of layered coding, for example for multiview coding, qualityscalability, spatial scalability, and for multiview video plus depthcoding.

Embodiments of the present invention may be implemented in software,hardware, application logic or a combination of software, hardware andapplication logic. In an example embodiment, the application logic,software or an instruction set is maintained on any one of variousconventional computer-readable media. In the context of this document, a“computer-readable medium” may be any media or means that can contain,store, communicate, propagate or transport the instructions for use byor in connection with an instruction execution system, apparatus, ordevice, such as a computer, with one example of a computer described anddepicted in FIGS. 1 and 2. A computer-readable medium may comprise acomputer-readable storage medium that may be any media or means that cancontain or store the instructions for use by or in connection with aninstruction execution system, apparatus, or device, such as a computer.

If desired, the different functions discussed herein may be performed ina different order and/or concurrently with each other. Furthermore, ifdesired, one or more of the above-described functions may be optional ormay be combined.

Although the above examples describe embodiments of the inventionoperating within a codec within an electronic device, it would beappreciated that the invention as described below may be implemented aspart of any video codec. Thus, for example, embodiments of the inventionmay be implemented in a video codec which may implement video codingover fixed or wired communication paths.

Thus, user equipment may comprise a video codec such as those describedin embodiments of the invention above. It shall be appreciated that theterm user equipment is intended to cover any suitable type of wirelessuser equipment, such as mobile telephones, portable data processingdevices or portable web browsers.

Furthermore elements of a public land mobile network (PLMN) may alsocomprise video codecs as described above.

In general, the various embodiments of the invention may be implementedin hardware or special purpose circuits, software, logic or anycombination thereof. For example, some aspects may be implemented inhardware, while other aspects may be implemented in firmware or softwarewhich may be executed by a controller, microprocessor or other computingdevice, although the invention is not limited thereto. While variousaspects of the invention may be illustrated and described as blockdiagrams, flow charts, or using some other pictorial representation, itis well understood that these blocks, apparatuses, systems, techniquesor methods described herein may be implemented in, as non-limitingexamples, hardware, software, firmware, special purpose circuits orlogic, general purpose hardware or controller or other computingdevices, or some combination thereof.

The embodiments of this invention may be implemented by computersoftware executable by a data processor of the mobile device, such as inthe processor entity, or by hardware, or by a combination of softwareand hardware. Further in this regard it should be noted that any blocksof the logic flow as in the Figures may represent program steps, orinterconnected logic circuits, blocks and functions, or a combination ofprogram steps and logic circuits, blocks and functions. The software maybe stored on such physical media as memory chips, or memory blocksimplemented within the processor, magnetic media such as hard disk orfloppy disks, and optical media such as for example DVD and the datavariants thereof, CD.

The various embodiments of the invention can be implemented with thehelp of computer program code that resides in a memory and causes therelevant apparatuses to carry out the invention. For example, a terminaldevice may comprise circuitry and electronics for handling, receivingand transmitting data, computer program code in a memory, and aprocessor that, when running the computer program code, causes theterminal device to carry out the features of an embodiment. Yet further,a network device may comprise circuitry and electronics for handling,receiving and transmitting data, computer program code in a memory, anda processor that, when running the computer program code, causes thenetwork device to carry out the features of an embodiment.

The memory may be of any type suitable to the local technicalenvironment and may be implemented using any suitable data storagetechnology, such as semiconductor based memory devices, magnetic memorydevices and systems, optical memory devices and systems, fixed memoryand removable memory. The data processors may be of any type suitable tothe local technical environment, and may include one or more of generalpurpose computers, special purpose computers, microprocessors, digitalsignal processors (DSPs) and processors based on multi core processorarchitecture, as non-limiting examples.

Embodiments of the inventions may be practiced in various componentssuch as integrated circuit modules. The design of integrated circuits isby and large a highly automated process. Complex and powerful softwaretools are available for converting a logic level design into asemiconductor circuit design ready to be etched and formed on asemiconductor substrate.

Programs, such as those provided by Synopsys Inc., of Mountain View,Calif. and Cadence Design, of San Jose, Calif. automatically routeconductors and locate components on a semiconductor chip using wellestablished rules of design as well as libraries of pre stored designmodules. Once the design for a semiconductor circuit has been completed,the resultant design, in a standardized electronic format (e.g., Opus,GDSII, or the like) may be transmitted to a semiconductor fabricationfacility or “fab” for fabrication.

The foregoing description has provided by way of exemplary andnon-limiting examples a full and informative description of theexemplary embodiment of this invention. However, various modificationsand adaptations may become apparent to those skilled in the relevantarts in view of the foregoing description, when read in conjunction withthe accompanying drawings and the appended claims. However, all such andsimilar modifications of the teachings of this invention will still fallwithin the scope of this invention.

The invention claimed is:
 1. A method comprising: encoding pictures intoa bitstream, the bitstream comprising at least two scalability layers,pictures being associated with access units and pictures beingassociated individually with one of the at least two scalability layers;indicating in the bitstream inter-layer prediction dependencies,indicative of direct reference layers, if any, of a first scalabilitylayer and indirect reference layers, if any, of the first scalabilitylayer, a direct reference layer being such that a picture associatedwith the direct reference layer is useable as a reference picture forprediction of a picture associated with the first scalability layer, andan indirect reference layer being such that a picture associated with anindirect reference layer is unusable as a reference picture forprediction of a picture associated with the first scalability layer butuseable as a reference picture for prediction of a picture associatedwith a direct or indirect reference layer of the first scalabilitylayer; selecting an earlier picture in decoding order as a basis forderiving picture order count (POC) related variables for a currentpicture based on a pre-defined algorithm, the current picture beingassociated with a current scalability layer, wherein the earlier pictureis the closest preceding picture, in decoding order, to the currentpicture among a set of pictures that are associated with the currentscalability layer or any direct or indirect reference layer of thecurrent scalability layer, and wherein the method further comprises as apart of or in connection with encoding the current picture into thebitstream: determining that POC values are to be reset; encoding intothe bitstream an indication of resetting POC values; and using theearlier picture in deriving a value decremented from POC values of allof a plurality of pictures in a decoded picture buffer.
 2. The methodaccording to claim 1, wherein the earlier picture is associated with aPOC value variable that can be represented as a sum of a mostsignificant part and a least significant part, wherein the leastsignificant part is represented by a fixed number of bits in a binaryrepresentation and the fixed number of least significant bits of themost significant part are equal to
 0. 3. The method according to claim1, the method further comprising: encoding at least one POC relatedsyntax element into the bitstream on the basis of the POC relatedvariables of said earlier picture defined as a POC reference picture. 4.The method according to claim 3, wherein the POC related syntax elementrepresents a fixed number of least significant bits of a POC value ofthe current picture; and the pre-defined algorithm comprisesinvestigating a difference of the value of the POC related syntaxelement and the least significant part of the POC value of the earlierpicture.
 5. The method according to claim 1, wherein pictures arecharacterized in one or more of the following characterizations: beingassociated individually with one of one or more temporal sub-layers,characterized in that prediction of a picture associated with aparticular temporal sub-layer is disabled from any picture associatedwith a temporal sub-layer higher than the particular temporal sub-layer,being classified individually as one of a sub-layer reference pictureand sub-layer non-reference picture, characterized in that prediction ofa picture associated with a first temporal sub-layer is enabled from asub-layer reference picture associated with the first temporalsub-layer, and prediction of a picture associated with a second temporalsub-layer is disabled from a sub-layer non-reference picture associatedwith the second temporal sub-layer, being classified individually as oneof a leading picture, an intra random access point (IRAP) picture, and atrailing picture, an IRAP picture enabling starting of decoding of ascalability layer which the IRAP picture is associated with, a leadingpicture following an associated IRAP picture in decoding order andpreceding the associated IRAP picture in output order, a trailingpicture following an associated IRAP picture in output order, beingclassified individually as one of a discardable picture and anon-discardable picture, a discardable picture characterized in that noother picture uses it as a reference picture for prediction, wherein themethod further comprises: determining POC reference picturecharacteristics according to said one or more of characterizations tomatch characteristics of pictures on which the current picture maydepend on in prediction; and selecting the set of pictures to comprisepictures fulfilling the POC reference picture characteristics.
 6. Themethod according to claim 5, further comprising: determining POCreference picture characteristics to match pictures that are associatedwith the lowest temporal sub-layer, are sub-layer reference pictures,are either intra random access point pictures or trailing pictures, andare non-discardable pictures.
 7. An apparatus comprising at least oneprocessor and at least one memory, said at least one memory stored withcode thereon, which when executed by said at least one processor, causesan apparatus to perform at least the following: encode pictures into abitstream, the bitstream comprising at least two scalability layers,pictures being associated with access units and pictures beingassociated individually with one of the at least two scalability layers;indicate in the bitstream inter-layer prediction dependencies,indicative of direct reference layers, if any, of a first scalabilitylayer and indirect reference layers, if any, of the first scalabilitylayer, a direct reference layer being such that a picture associatedwith the direct reference layer is useable as a reference picture forprediction of a picture associated with the first scalability layer, andan indirect reference layer being such that a picture associated with anindirect reference layer is unusable as a reference picture forprediction of a picture associated with the first scalability layer butuseable as a reference picture for prediction of a picture associatedwith a direct or indirect reference layer of the first scalabilitylayer; select an earlier picture in decoding order as a basis forderiving picture order count (POC) related variables for a currentpicture based on a pre-defined algorithm, the current picture beingassociated with a current scalability layer, wherein the earlier pictureis the closest preceding picture, in decoding order, to the currentpicture among a set of pictures that are associated with the currentscalability layer or any direct or indirect reference layer of thecurrent scalability layer, and wherein the apparatus is further caused,as a part of or in connection with encoding the current picture into thebitstream, to perform: determine that POC values are to be reset; encodeinto the bitstream an indication of resetting POC values; and use theearlier picture in deriving a value decremented from POC values of allof a plurality of pictures in a decoded picture buffer.
 8. A computerprogram product embodied on a non-transitory computer readable medium,comprising computer program code configured to, when executed on atleast one processor, cause an apparatus or a system to: encode picturesinto a bitstream, the bitstream comprising at least two scalabilitylayers, pictures being associated with access units and pictures beingassociated individually with one of the at least two scalability layers;indicate in the bitstream inter-layer prediction dependencies,indicative of direct reference layers, if any, of a first scalabilitylayer and indirect reference layers, if any, of the first scalabilitylayer, a direct reference layer being such that a picture associatedwith the direct reference layer is useable as a reference picture forprediction of a picture associated with the first scalability layer, andan indirect reference layer being such that a picture associated with anindirect reference layer is unusable as a reference picture forprediction of a picture associated with the first scalability layer butuseable as a reference picture for prediction of a picture associatedwith a direct or indirect reference layer of the first scalabilitylayer; select an earlier picture in decoding order as a basis forderiving picture order count (POC) related variables for a currentpicture based on a pre-defined algorithm, the current picture beingassociated with a current scalability layer, wherein the earlier pictureis the closest preceding picture, in decoding order, to the currentpicture among a set of pictures that are associated with the currentscalability layer or any direct or indirect reference layer of thecurrent scalability layer, and wherein the apparatus or the system isfurther caused, as a part of or in connection with encoding the currentpicture into the bitstream, to: determine that POC values are to bereset; encode into the bitstream an indication of resetting POC values;and use the earlier picture in deriving a value decremented from POCvalues of all of a plurality of pictures in a decoded picture buffer. 9.A video encoder configured for: encoding pictures into a bitstream, thebitstream comprising at least two scalability layers, pictures beingassociated with access units and pictures being associated individuallywith one of the at least two scalability layers; indicating in thebitstream inter-layer prediction dependencies, indicative of directreference layers, if any, of a first scalability layer and indirectreference layers, if any, of the first scalability layer, a directreference layer being such that a picture associated with the directreference layer is useable as a reference picture for prediction of apicture associated with the first scalability layer, and an indirectreference layer being such that a picture associated with an indirectreference layer is unusable as a reference picture for prediction of apicture associated with the first scalability layer but useable as areference picture for prediction of a picture associated with a director indirect reference layer of the first scalability layer; selecting anearlier picture in decoding order as a basis for deriving picture ordercount (POC) related variables for a current picture based on apre-defined algorithm, the current picture being associated with acurrent scalability layer, wherein the earlier picture is the closestpreceding picture, in decoding order, to the current picture among a setof pictures that are associated with the current scalability layer orany direct or indirect reference layer of the current scalability layer,and wherein the video encoder is further configured, as a part of or inconnection with encoding the current picture into the bitstream, for:determining that POC values are to be reset; encoding into the bitstreaman indication of resetting POC values; and using the earlier picture inderiving a value decremented from POC values of all of a plurality ofpictures in a decoded picture buffer.
 10. A method comprising: decodingpictures from a bitstream, the bitstream comprising at least twoscalability layers, pictures being associated with access units andpictures being associated individually with one of the at least twoscalability layers; obtaining an indication from the bitstream relatingto inter-layer prediction dependencies, indicative of direct referencelayers, if any, of a first scalability layer and indirect referencelayers, if any, of the first scalability layer, a direct reference layerbeing such that a picture associated with the direct reference layer isuseable as a reference picture for prediction of a picture associatedwith the first scalability layer, and an indirect reference layer beingsuch that a picture associated with an indirect reference layer isunusable as a reference picture for prediction of a picture associatedwith the first scalability layer but useable as a reference picture forprediction of a picture associated with a direct or indirect referencelayer of the first scalability layer; selecting an earlier picture indecoding order as a basis for deriving picture order count (POC) relatedvariables for a current picture based on a pre-defined algorithm, thecurrent picture being associated with a current scalability layer,wherein the earlier picture is the closest preceding picture, indecoding order, to the current picture among a set of pictures that areassociated with the current scalability layer or any direct or indirectreference layer of the current scalability layer, and wherein the methodfurther comprises as a part of or in connection with decoding thecurrent picture from the bitstream: determining that POC values are tobe reset; decoding into the bitstream an indication of resetting POCvalues; and using the earlier picture in deriving a value decrementedfrom POC values of all of the plurality of pictures in a decoded picturebuffer.
 11. The method according to claim 10, wherein the earlierpicture is associated with a POC value variable that can be representedas a sum of a most significant part and a least significant part,wherein the least significant part is represented by a fixed number ofbits in a binary representation and the fixed number of leastsignificant bits of the most significant part are equal to
 0. 12. Themethod according to claim 10, the method further comprising: decoding atleast one POC related syntax element from the bitstream; and decodingPOC related variables of the current picture on the basis of the decodedat least one POC related syntax element and the POC related variables ofsaid earlier picture defined as a POC reference picture.
 13. The methodaccording to claim 12, wherein the POC related syntax element representsa fixed number of least significant bits of a POC value of the currentpicture; and the pre-defined algorithm comprises investigating adifference of the value of the POC related syntax element and the leastsignificant part of the POC value of the earlier picture.
 14. The methodaccording to claim 10, wherein pictures are characterized in one or moreof the following characterizations: being associated individually withone of one or more temporal sub-layers, characterized in that predictionof a picture associated with a particular temporal sub-layer is disabledfrom any picture associated with a temporal sub-layer higher than theparticular temporal sub-layer, being classified individually as one of asub-layer reference picture and sub-layer non-reference picture,characterized in that prediction of a picture associated with a firsttemporal sub-layer is enabled from a sub-layer reference pictureassociated with the first temporal sub-layer, and prediction of apicture associated with a second temporal sub-layer is disabled from asub-layer non-reference picture associated with the second temporalsub-layer, being classified individually as one of a leading picture, anintra random access point (IRAP) picture, and a trailing picture, anIRAP picture enabling starting of decoding of a scalability layer whichthe IRAP picture is associated with, a leading picture following anassociated IRAP picture in decoding order and preceding the associatedIRAP picture in output order, a trailing picture following an associatedIRAP picture in output order, being classified individually as one of adiscardable picture and a non-discardable picture, a discardable picturecharacterized in that no other picture uses it as a reference picturefor prediction, wherein the method further comprises: determining POCreference picture characteristics according to said one or more ofcharacterizations to match characteristics of pictures on which thecurrent picture may depend on in prediction; and selecting the set ofpictures to comprise pictures fulfilling the POC reference picturecharacteristics.
 15. The method according to claim 14, furthercomprising: determining POC reference picture characteristics to matchpictures that are associated with the lowest temporal sub-layer, aresub-layer reference pictures, are either intra random access pointpictures or trailing pictures, and are non-discardable pictures.
 16. Anapparatus comprising at least one processor and at least one memory,said at least one memory stored with code thereon, which when executedby said at least one processor, causes an apparatus to perform at leastthe following: decoding pictures from a bitstream, the bitstreamcomprising at least two scalability layers, pictures being associatedwith access units and pictures being associated individually with one ofthe at least two scalability layers; obtaining an indication from thebitstream relating to inter-layer prediction dependencies, indicative ofdirect reference layers, if any, of a first scalability layer andindirect reference layers, if any, of the first scalability layer, adirect reference layer being such that a picture associated with thedirect reference layer is useable as a reference picture for predictionof a picture associated with the first scalability layer, and anindirect reference layer being such that a picture associated with anindirect reference layer is unusable as a reference picture forprediction of a picture associated with the first scalability layer butuseable as a reference picture for prediction of a picture associatedwith a direct or indirect reference layer of the first scalabilitylayer; selecting an earlier picture in decoding order as a basis forderiving picture order count (POC) related variables for a currentpicture based on a pre-defined algorithm, the current picture beingassociated with a current scalability layer, wherein the earlier pictureis the closest preceding picture, in decoding order, to the currentpicture among a set of pictures that are associated with the currentscalability layer or any direct or indirect reference layer of thecurrent scalability layer, and wherein the apparatus is further causedto, as a part of or in connection with decoding the current picture fromthe bitstream, perform: determining that POC values are to be reset;decoding into the bitstream an indication of resetting POC values; andusing the earlier picture in deriving a value decremented from POCvalues of all of the plurality of pictures in a decoded picture buffer.17. A computer program product embodied on a non-transitory computerreadable medium, comprising computer program code configured to, whenexecuted on at least one processor, cause an apparatus or a system to:decode pictures from a bitstream, the bitstream comprising at least twoscalability layers, pictures being associated with access units andpictures being associated individually with one of the at least twoscalability layers; obtain an indication from the bitstream relating tointer-layer prediction dependencies, indicative of direct referencelayers, if any, of a first scalability layer and indirect referencelayers, if any, of the first scalability layer, a direct reference layerbeing such that a picture associated with the direct reference layer isuseable as a reference picture for prediction of a picture associatedwith the first scalability layer, and an indirect reference layer beingsuch that a picture associated with an indirect reference layer isunusable as a reference picture for prediction of a picture associatedwith the first scalability layer but useable as a reference picture forprediction of a picture associated with a direct or indirect referencelayer of the first scalability layer; select an earlier picture indecoding order as a basis for deriving picture order count (POC) relatedvariables for a current picture based on a pre-defined algorithm, thecurrent picture being associated with a current scalability layer,wherein the earlier picture is the closest preceding picture, indecoding order, to the current picture among a set of pictures that areassociated with the current scalability layer or any direct or indirectreference layer of the current scalability layer, and wherein theapparatus or the system is further caused, as a part of or in connectionwith decoding the current picture from the bitstream, to: determine thatPOC values are to be reset; decode into the bitstream an indication ofresetting POC values; and use the earlier picture in deriving a valuedecremented from POC values of all of the plurality of pictures in adecoded picture buffer.
 18. A video decoder configured for: decodingpictures from a bitstream, the bitstream comprising at least twoscalability layers, pictures being associated with access units andpictures being associated individually with one of the at least twoscalability layers; obtaining an indication from the bitstream relatingto inter-layer prediction dependencies, indicative of direct referencelayers, if any, of a first scalability layer and indirect referencelayers, if any, of the first scalability layer, a direct reference layerbeing such that a picture associated with the direct reference layer isuseable as a reference picture for prediction of a picture associatedwith the first scalability layer, and an indirect reference layer beingsuch that a picture associated with an indirect reference layer isunusable as a reference picture for prediction of a picture associatedwith the first scalability layer but useable as a reference picture forprediction of a picture associated with a direct or indirect referencelayer of the first scalability layer; selecting an earlier picture indecoding order as a basis for deriving picture order count (POC) relatedvariables for a current picture based on a pre-defined algorithm, thecurrent picture being associated with a current scalability layer,wherein the earlier picture is the closest preceding picture, indecoding order, to the current picture among a set of pictures that areassociated with the current scalability layer or any direct or indirectreference layer of the current scalability layer, and wherein the videodecoder is further configured as a part of or in connection withdecoding the current picture from the bitstream, for: determining thatPOC values are to be reset; decoding into the bitstream an indication ofresetting POC values; and using the earlier picture in deriving a valuedecremented from POC values of all of the plurality of pictures in adecoded picture buffer.